Jul 02 19:05:56.195555 ip-10-0-133-147 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 02 19:05:56.195566 ip-10-0-133-147 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 02 19:05:56.195573 ip-10-0-133-147 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 02 19:05:56.195867 ip-10-0-133-147 systemd[1]: Failed to start Kubernetes Kubelet. Jul 02 19:06:06.314924 ip-10-0-133-147 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 02 19:06:06.314954 ip-10-0-133-147 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 3d90c9950bd647d087f891268adbf98e -- Jul 02 19:08:12.104991 ip-10-0-133-147 systemd[1]: Starting Kubernetes Kubelet... Jul 02 19:08:12.503581 ip-10-0-133-147 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 19:08:12.503581 ip-10-0-133-147 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 02 19:08:12.503581 ip-10-0-133-147 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 19:08:12.503581 ip-10-0-133-147 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 02 19:08:12.503581 ip-10-0-133-147 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 19:08:12.505332 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.505254 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 02 19:08:12.511044 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511029 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:08:12.511044 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511044 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:08:12.511109 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511048 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:08:12.511109 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511051 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:08:12.511109 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511055 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:08:12.511109 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511058 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:08:12.511109 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511063 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:08:12.511109 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511067 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:08:12.511109 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511070 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:08:12.511109 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511072 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:08:12.511109 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511075 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:08:12.511109 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511078 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:08:12.511109 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511081 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:08:12.511109 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511084 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:08:12.511109 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511087 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:08:12.511109 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511089 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:08:12.511109 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511092 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:08:12.511109 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511095 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:08:12.511109 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511098 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:08:12.511109 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511100 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:08:12.511109 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511104 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:08:12.511109 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511106 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:08:12.511572 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511109 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:08:12.511572 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511112 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:08:12.511572 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511115 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:08:12.511572 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511117 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:08:12.511572 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511120 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:08:12.511572 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511123 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:08:12.511572 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511126 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:08:12.511572 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511129 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:08:12.511572 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511132 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:08:12.511572 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511135 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:08:12.511572 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511138 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:08:12.511572 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511140 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:08:12.511572 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511143 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:08:12.511572 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511146 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:08:12.511572 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511148 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:08:12.511572 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511151 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:08:12.511572 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511153 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:08:12.511572 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511156 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:08:12.511572 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511158 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:08:12.511572 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511161 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:08:12.512069 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511164 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:08:12.512069 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511166 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:08:12.512069 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511169 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:08:12.512069 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511172 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:08:12.512069 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511176 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:08:12.512069 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511180 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:08:12.512069 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511182 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:08:12.512069 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511184 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:08:12.512069 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511187 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:08:12.512069 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511189 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:08:12.512069 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511192 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:08:12.512069 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511194 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:08:12.512069 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511196 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:08:12.512069 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511200 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:08:12.512069 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511202 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:08:12.512069 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511205 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:08:12.512069 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511208 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:08:12.512069 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511210 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:08:12.512069 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511213 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:08:12.512069 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511215 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:08:12.512550 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511218 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:08:12.512550 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511221 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:08:12.512550 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511223 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:08:12.512550 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511226 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:08:12.512550 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511228 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:08:12.512550 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511231 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:08:12.512550 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511233 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:08:12.512550 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511236 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:08:12.512550 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511238 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:08:12.512550 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511241 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:08:12.512550 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511243 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:08:12.512550 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511245 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:08:12.512550 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511249 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:08:12.512550 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511252 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:08:12.512550 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511254 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:08:12.512550 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511257 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:08:12.512550 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511259 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:08:12.512550 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511262 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:08:12.512550 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511264 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:08:12.512550 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511267 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:08:12.513047 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511269 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:08:12.513047 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511273 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:08:12.513047 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511277 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:08:12.513047 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511280 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:08:12.513047 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511282 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:08:12.513047 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511286 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:08:12.513047 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511289 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:08:12.513047 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511292 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:08:12.513047 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511294 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:08:12.513047 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511297 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:08:12.513047 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511299 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:08:12.513047 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511302 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:08:12.513047 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511305 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:08:12.513047 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511307 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:08:12.513047 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511310 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:08:12.513047 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511312 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:08:12.513047 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.511315 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:08:12.513047 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512080 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:08:12.513047 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512087 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:08:12.513499 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512090 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:08:12.513499 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512093 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:08:12.513499 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512096 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:08:12.513499 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512099 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:08:12.513499 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512101 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:08:12.513499 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512104 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:08:12.513499 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512107 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:08:12.513499 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512110 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:08:12.513499 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512113 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:08:12.513499 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512116 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:08:12.513499 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512120 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:08:12.513499 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512122 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:08:12.513499 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512125 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:08:12.513499 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512128 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:08:12.513499 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512130 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:08:12.513499 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512133 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:08:12.513499 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512136 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:08:12.513499 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512139 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:08:12.513499 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512141 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:08:12.513985 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512144 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:08:12.513985 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512146 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:08:12.513985 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512149 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:08:12.513985 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512151 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:08:12.513985 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512154 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:08:12.513985 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512157 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:08:12.513985 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512160 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:08:12.513985 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512162 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:08:12.513985 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512165 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:08:12.513985 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512168 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:08:12.513985 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512170 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:08:12.513985 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512173 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:08:12.513985 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512175 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:08:12.513985 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512177 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:08:12.513985 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512180 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:08:12.513985 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512183 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:08:12.513985 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512186 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:08:12.513985 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512188 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:08:12.513985 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512191 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:08:12.513985 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512193 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:08:12.514472 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512196 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:08:12.514472 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512199 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:08:12.514472 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512201 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:08:12.514472 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512203 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:08:12.514472 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512207 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:08:12.514472 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512211 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:08:12.514472 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512215 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:08:12.514472 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512218 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:08:12.514472 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512221 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:08:12.514472 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512224 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:08:12.514472 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512226 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:08:12.514472 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512229 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:08:12.514472 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512232 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:08:12.514472 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512235 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:08:12.514472 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512237 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:08:12.514472 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512240 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:08:12.514472 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512242 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:08:12.514472 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512246 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:08:12.514472 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512249 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:08:12.515003 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512252 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:08:12.515003 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512255 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:08:12.515003 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512257 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:08:12.515003 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512260 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:08:12.515003 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512263 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:08:12.515003 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512265 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:08:12.515003 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512268 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:08:12.515003 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512270 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:08:12.515003 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512272 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:08:12.515003 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512274 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:08:12.515003 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512277 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:08:12.515003 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512279 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:08:12.515003 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512282 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:08:12.515003 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512284 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:08:12.515003 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512287 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:08:12.515003 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512290 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:08:12.515003 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512292 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:08:12.515003 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512295 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:08:12.515003 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512297 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:08:12.515003 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512302 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:08:12.515481 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512304 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:08:12.515481 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512307 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:08:12.515481 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512309 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:08:12.515481 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512312 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:08:12.515481 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512315 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:08:12.515481 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512317 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:08:12.515481 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512320 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:08:12.515481 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512322 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:08:12.515481 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512325 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:08:12.515481 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512327 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:08:12.515481 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512329 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:08:12.515481 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512332 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:08:12.515481 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512334 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:08:12.515481 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512337 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:08:12.515481 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512339 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:08:12.515481 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512341 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:08:12.515481 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512344 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:08:12.515481 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512346 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:08:12.515481 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512349 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:08:12.515481 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512763 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:08:12.515989 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512769 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:08:12.515989 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512772 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:08:12.515989 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512775 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:08:12.515989 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512777 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:08:12.515989 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512780 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:08:12.515989 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512783 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:08:12.515989 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512785 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:08:12.515989 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512789 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:08:12.515989 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512792 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:08:12.515989 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512795 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:08:12.515989 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512797 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:08:12.515989 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512803 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:08:12.515989 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512805 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:08:12.515989 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512809 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:08:12.515989 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512811 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:08:12.515989 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512813 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:08:12.515989 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512816 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:08:12.515989 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512819 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:08:12.515989 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512821 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:08:12.515989 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512824 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:08:12.516493 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512826 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:08:12.516493 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512829 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:08:12.516493 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512832 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:08:12.516493 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512834 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:08:12.516493 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512836 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:08:12.516493 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512839 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:08:12.516493 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512842 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:08:12.516493 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512844 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:08:12.516493 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512846 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:08:12.516493 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512849 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:08:12.516493 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512852 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:08:12.516493 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512854 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:08:12.516493 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512856 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:08:12.516493 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512859 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:08:12.516493 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512861 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:08:12.516493 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512866 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:08:12.516493 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512870 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:08:12.516493 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512873 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:08:12.516493 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512877 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:08:12.516493 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512880 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:08:12.516996 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512884 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:08:12.516996 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512887 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:08:12.516996 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512890 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:08:12.516996 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512894 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:08:12.516996 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512897 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:08:12.516996 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512899 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:08:12.516996 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512902 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:08:12.516996 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512904 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:08:12.516996 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512907 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:08:12.516996 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512909 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:08:12.516996 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512913 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:08:12.516996 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512917 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:08:12.516996 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512920 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:08:12.516996 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512923 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:08:12.516996 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512925 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:08:12.516996 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512928 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:08:12.516996 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512931 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:08:12.516996 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512933 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:08:12.516996 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512936 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:08:12.517450 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512940 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:08:12.517450 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512943 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:08:12.517450 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512946 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:08:12.517450 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512949 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:08:12.517450 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512952 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:08:12.517450 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512954 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:08:12.517450 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512957 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:08:12.517450 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512959 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:08:12.517450 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512962 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:08:12.517450 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512965 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:08:12.517450 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512967 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:08:12.517450 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512970 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:08:12.517450 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512972 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:08:12.517450 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512975 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:08:12.517450 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512978 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:08:12.517450 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512980 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:08:12.517450 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512985 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:08:12.517450 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512988 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:08:12.517450 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512990 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:08:12.517450 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512993 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:08:12.517993 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512995 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:08:12.517993 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.512998 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:08:12.517993 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513000 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:08:12.517993 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513003 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:08:12.517993 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513005 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:08:12.517993 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513007 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:08:12.517993 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513010 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:08:12.517993 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513012 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:08:12.517993 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513015 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:08:12.517993 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513017 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:08:12.517993 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513019 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:08:12.517993 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513022 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:08:12.517993 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513024 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:08:12.517993 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513027 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:08:12.517993 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513030 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:08:12.517993 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513032 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:08:12.517993 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513035 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:08:12.517993 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513037 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:08:12.517993 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513040 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:08:12.517993 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513115 2562 flags.go:64] FLAG: --address="0.0.0.0" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513125 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513133 2562 flags.go:64] FLAG: --anonymous-auth="true" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513139 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513145 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513149 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513153 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513157 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513161 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513166 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513170 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513173 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513176 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513179 2562 flags.go:64] FLAG: --cgroup-root="" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513182 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513184 2562 flags.go:64] FLAG: --client-ca-file="" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513187 2562 flags.go:64] FLAG: --cloud-provider="external" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513190 2562 flags.go:64] FLAG: --cluster-dns="[]" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513195 2562 flags.go:64] FLAG: --cluster-domain="" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513198 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513200 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513203 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513207 2562 flags.go:64] FLAG: --container-log-max-files="5" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513211 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 02 19:08:12.518463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513214 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513217 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513220 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513223 2562 flags.go:64] FLAG: --contention-profiling="false" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513226 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513229 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513232 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513235 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513239 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513242 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513244 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513247 2562 flags.go:64] FLAG: --enable-load-reader="false" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513250 2562 flags.go:64] FLAG: --enable-server="true" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513253 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513257 2562 flags.go:64] FLAG: --event-burst="100" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513259 2562 flags.go:64] FLAG: --event-qps="50" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513262 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513268 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513271 2562 flags.go:64] FLAG: --eviction-hard="" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513274 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513277 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513281 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513284 2562 flags.go:64] FLAG: --eviction-soft="" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513287 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513290 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 02 19:08:12.519078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513293 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513295 2562 flags.go:64] FLAG: --experimental-mounter-path="" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513299 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513302 2562 flags.go:64] FLAG: --fail-swap-on="true" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513305 2562 flags.go:64] FLAG: --feature-gates="" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513309 2562 flags.go:64] FLAG: --file-check-frequency="20s" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513312 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513315 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513318 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513321 2562 flags.go:64] FLAG: --healthz-port="10248" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513324 2562 flags.go:64] FLAG: --help="false" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513326 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-133-147.ec2.internal" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513330 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513332 2562 flags.go:64] FLAG: --http-check-frequency="20s" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513335 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513338 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513342 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513345 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513347 2562 flags.go:64] FLAG: --image-service-endpoint="" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513350 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513353 2562 flags.go:64] FLAG: --kube-api-burst="100" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513356 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513359 2562 flags.go:64] FLAG: --kube-api-qps="50" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513361 2562 flags.go:64] FLAG: --kube-reserved="" Jul 02 19:08:12.519712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513367 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513371 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513374 2562 flags.go:64] FLAG: --kubelet-cgroups="" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513377 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513380 2562 flags.go:64] FLAG: --lock-file="" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513382 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513385 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513388 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513393 2562 flags.go:64] FLAG: --log-json-split-stream="false" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513396 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513399 2562 flags.go:64] FLAG: --log-text-split-stream="false" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513402 2562 flags.go:64] FLAG: --logging-format="text" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513405 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513408 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513411 2562 flags.go:64] FLAG: --manifest-url="" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513414 2562 flags.go:64] FLAG: --manifest-url-header="" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513418 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513421 2562 flags.go:64] FLAG: --max-open-files="1000000" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513425 2562 flags.go:64] FLAG: --max-pods="110" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513428 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513431 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513434 2562 flags.go:64] FLAG: --memory-manager-policy="None" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513437 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513440 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513443 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 02 19:08:12.520663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513446 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513452 2562 flags.go:64] FLAG: --node-status-max-images="50" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513455 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513458 2562 flags.go:64] FLAG: --oom-score-adj="-999" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513461 2562 flags.go:64] FLAG: --pod-cidr="" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513464 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513470 2562 flags.go:64] FLAG: --pod-manifest-path="" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513474 2562 flags.go:64] FLAG: --pod-max-pids="-1" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513477 2562 flags.go:64] FLAG: --pods-per-core="0" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513480 2562 flags.go:64] FLAG: --port="10250" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513483 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513486 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09bab2a88578fb8d4" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513489 2562 flags.go:64] FLAG: --qos-reserved="" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513492 2562 flags.go:64] FLAG: --read-only-port="10255" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513495 2562 flags.go:64] FLAG: --register-node="true" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513498 2562 flags.go:64] FLAG: --register-with-taints="" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513501 2562 flags.go:64] FLAG: --registry-burst="10" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513504 2562 flags.go:64] FLAG: --registry-qps="5" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513507 2562 flags.go:64] FLAG: --reserved-cpus="" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513510 2562 flags.go:64] FLAG: --reserved-memory="" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513514 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513517 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513519 2562 flags.go:64] FLAG: --rotate-certificates="false" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513522 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513525 2562 flags.go:64] FLAG: --runonce="false" Jul 02 19:08:12.521714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513528 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513531 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513534 2562 flags.go:64] FLAG: --seccomp-default="false" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513536 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513539 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513542 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513546 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513549 2562 flags.go:64] FLAG: --storage-driver-password="root" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513551 2562 flags.go:64] FLAG: --storage-driver-secure="false" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513554 2562 flags.go:64] FLAG: --storage-driver-table="stats" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513557 2562 flags.go:64] FLAG: --storage-driver-user="root" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513560 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513562 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513565 2562 flags.go:64] FLAG: --system-cgroups="" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513569 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513575 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513578 2562 flags.go:64] FLAG: --tls-cert-file="" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513581 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513587 2562 flags.go:64] FLAG: --tls-min-version="" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513606 2562 flags.go:64] FLAG: --tls-private-key-file="" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513609 2562 flags.go:64] FLAG: --topology-manager-policy="none" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513612 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513615 2562 flags.go:64] FLAG: --topology-manager-scope="container" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513618 2562 flags.go:64] FLAG: --v="2" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513622 2562 flags.go:64] FLAG: --version="false" Jul 02 19:08:12.522638 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513626 2562 flags.go:64] FLAG: --vmodule="" Jul 02 19:08:12.523240 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513629 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 02 19:08:12.523240 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.513633 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 02 19:08:12.523240 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513779 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:08:12.523240 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513784 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:08:12.523240 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513786 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:08:12.523240 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513789 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:08:12.523240 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513792 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:08:12.523240 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513795 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:08:12.523240 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513798 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:08:12.523240 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513801 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:08:12.523240 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513804 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:08:12.523240 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513806 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:08:12.523240 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513809 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:08:12.523240 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513812 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:08:12.523240 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513814 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:08:12.523240 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513817 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:08:12.523240 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513820 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:08:12.523240 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513822 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:08:12.523240 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513825 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:08:12.523240 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513827 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:08:12.523782 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513830 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:08:12.523782 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513834 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:08:12.523782 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513837 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:08:12.523782 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513839 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:08:12.523782 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513842 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:08:12.523782 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513845 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:08:12.523782 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513847 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:08:12.523782 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513850 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:08:12.523782 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513852 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:08:12.523782 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513855 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:08:12.523782 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513857 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:08:12.523782 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513860 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:08:12.523782 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513863 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:08:12.523782 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513865 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:08:12.523782 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513868 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:08:12.523782 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513870 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:08:12.523782 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513872 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:08:12.523782 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513875 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:08:12.523782 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513877 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:08:12.523782 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513880 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:08:12.524266 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513882 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:08:12.524266 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513885 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:08:12.524266 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513887 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:08:12.524266 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513890 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:08:12.524266 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513892 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:08:12.524266 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513895 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:08:12.524266 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513897 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:08:12.524266 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513900 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:08:12.524266 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513902 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:08:12.524266 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513905 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:08:12.524266 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513907 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:08:12.524266 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513910 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:08:12.524266 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513912 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:08:12.524266 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513916 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:08:12.524266 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513919 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:08:12.524266 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513921 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:08:12.524266 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513925 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:08:12.524266 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513929 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:08:12.524266 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513932 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:08:12.524737 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513935 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:08:12.524737 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513938 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:08:12.524737 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513940 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:08:12.524737 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513943 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:08:12.524737 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513946 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:08:12.524737 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513949 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:08:12.524737 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513952 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:08:12.524737 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513954 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:08:12.524737 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513958 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:08:12.524737 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513961 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:08:12.524737 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513964 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:08:12.524737 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513967 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:08:12.524737 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513970 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:08:12.524737 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513972 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:08:12.524737 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513975 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:08:12.524737 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513977 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:08:12.524737 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513980 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:08:12.524737 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513982 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:08:12.524737 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513984 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:08:12.525196 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513987 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:08:12.525196 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513990 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:08:12.525196 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513993 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:08:12.525196 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.513997 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:08:12.525196 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.514000 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:08:12.525196 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.514003 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:08:12.525196 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.514006 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:08:12.525196 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.514009 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:08:12.525196 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.514012 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:08:12.525196 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.514014 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:08:12.525196 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.514017 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:08:12.525196 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.514019 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:08:12.525196 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.514022 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:08:12.525196 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.514025 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:08:12.525196 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.514027 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:08:12.525196 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.514030 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:08:12.525196 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.514032 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:08:12.525196 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.514035 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:08:12.525196 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.514037 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:08:12.525196 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.514040 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:08:12.525759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.514043 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:08:12.525759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.514045 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:08:12.525759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.514047 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:08:12.525759 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.514686 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 19:08:12.525759 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.523154 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 02 19:08:12.525759 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.523168 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 02 19:08:12.525759 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.523829 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 02 19:08:12.525759 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.523840 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 02 19:08:12.525759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523894 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:08:12.525759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523899 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:08:12.525759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523903 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:08:12.525759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523906 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:08:12.525759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523909 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:08:12.525759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523912 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:08:12.526094 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523915 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:08:12.526094 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523917 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:08:12.526094 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523920 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:08:12.526094 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523922 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:08:12.526094 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523925 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:08:12.526094 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523927 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:08:12.526094 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523931 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:08:12.526094 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523933 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:08:12.526094 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523936 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:08:12.526094 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523939 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:08:12.526094 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523941 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:08:12.526094 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523944 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:08:12.526094 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523946 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:08:12.526094 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523949 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:08:12.526094 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523951 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:08:12.526094 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523954 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:08:12.526094 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523957 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:08:12.526094 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523959 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:08:12.526094 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523962 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:08:12.526094 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523968 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:08:12.526574 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523971 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:08:12.526574 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523973 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:08:12.526574 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523976 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:08:12.526574 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523979 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:08:12.526574 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523982 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:08:12.526574 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523985 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:08:12.526574 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523987 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:08:12.526574 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523989 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:08:12.526574 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523992 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:08:12.526574 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523994 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:08:12.526574 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523997 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:08:12.526574 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.523999 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:08:12.526574 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524002 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:08:12.526574 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524005 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:08:12.526574 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524007 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:08:12.526574 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524010 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:08:12.526574 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524012 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:08:12.526574 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524014 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:08:12.526574 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524017 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:08:12.526574 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524019 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:08:12.527097 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524022 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:08:12.527097 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524024 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:08:12.527097 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524026 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:08:12.527097 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524029 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:08:12.527097 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524031 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:08:12.527097 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524033 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:08:12.527097 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524036 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:08:12.527097 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524038 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:08:12.527097 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524041 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:08:12.527097 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524043 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:08:12.527097 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524046 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:08:12.527097 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524049 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:08:12.527097 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524052 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:08:12.527097 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524054 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:08:12.527097 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524057 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:08:12.527097 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524060 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:08:12.527097 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524063 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:08:12.527097 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524065 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:08:12.527097 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524068 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:08:12.527097 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524070 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:08:12.527573 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524073 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:08:12.527573 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524077 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:08:12.527573 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524080 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:08:12.527573 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524083 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:08:12.527573 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524087 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:08:12.527573 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524089 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:08:12.527573 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524092 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:08:12.527573 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524094 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:08:12.527573 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524097 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:08:12.527573 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524099 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:08:12.527573 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524102 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:08:12.527573 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524104 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:08:12.527573 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524107 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:08:12.527573 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524110 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:08:12.527573 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524112 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:08:12.527573 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524115 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:08:12.527573 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524117 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:08:12.527573 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524119 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:08:12.527573 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524122 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:08:12.528045 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524124 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:08:12.528045 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524127 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:08:12.528045 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524129 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:08:12.528045 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524131 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:08:12.528045 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524134 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:08:12.528045 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524136 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:08:12.528045 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524139 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:08:12.528045 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524141 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:08:12.528045 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524145 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:08:12.528045 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524149 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:08:12.528045 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524153 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:08:12.528045 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524157 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:08:12.528045 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524160 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:08:12.528045 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524162 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:08:12.528564 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.524167 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 19:08:12.528564 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524298 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:08:12.528564 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524303 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:08:12.528564 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524306 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:08:12.528564 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524308 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:08:12.528564 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524311 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:08:12.528564 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524313 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:08:12.528564 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524316 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:08:12.528564 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524319 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:08:12.528564 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524321 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:08:12.528564 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524324 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:08:12.528564 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524326 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:08:12.528564 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524329 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:08:12.528564 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524332 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:08:12.528564 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524334 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:08:12.529215 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524337 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:08:12.529215 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524339 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:08:12.529215 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524342 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:08:12.529215 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524344 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:08:12.529215 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524347 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:08:12.529215 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524349 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:08:12.529215 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524351 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:08:12.529215 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524354 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:08:12.529215 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524356 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:08:12.529215 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524360 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:08:12.529215 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524364 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:08:12.529215 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524367 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:08:12.529215 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524370 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:08:12.529215 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524373 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:08:12.529215 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524376 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:08:12.529215 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524378 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:08:12.529215 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524381 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:08:12.529215 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524383 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:08:12.529215 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524386 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:08:12.529215 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524388 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:08:12.529759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524391 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:08:12.529759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524394 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:08:12.529759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524396 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:08:12.529759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524398 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:08:12.529759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524401 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:08:12.529759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524403 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:08:12.529759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524407 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:08:12.529759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524410 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:08:12.529759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524412 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:08:12.529759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524415 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:08:12.529759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524418 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:08:12.529759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524421 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:08:12.529759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524424 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:08:12.529759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524426 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:08:12.529759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524429 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:08:12.529759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524431 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:08:12.529759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524434 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:08:12.529759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524437 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:08:12.529759 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524439 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:08:12.530261 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524442 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:08:12.530261 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524444 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:08:12.530261 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524447 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:08:12.530261 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524450 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:08:12.530261 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524453 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:08:12.530261 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524456 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:08:12.530261 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524458 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:08:12.530261 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524461 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:08:12.530261 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524463 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:08:12.530261 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524466 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:08:12.530261 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524468 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:08:12.530261 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524471 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:08:12.530261 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524473 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:08:12.530261 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524475 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:08:12.530261 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524478 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:08:12.530261 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524480 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:08:12.530261 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524483 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:08:12.530261 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524486 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:08:12.530261 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524489 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:08:12.530869 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524492 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:08:12.530869 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524495 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:08:12.530869 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524497 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:08:12.530869 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524499 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:08:12.530869 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524502 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:08:12.530869 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524504 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:08:12.530869 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524507 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:08:12.530869 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524510 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:08:12.530869 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524512 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:08:12.530869 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524514 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:08:12.530869 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524517 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:08:12.530869 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524520 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:08:12.530869 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524522 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:08:12.530869 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524528 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:08:12.530869 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524530 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:08:12.530869 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524533 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:08:12.530869 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524536 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:08:12.530869 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524538 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:08:12.530869 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524541 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:08:12.530869 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524543 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:08:12.530869 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524546 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:08:12.531377 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524548 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:08:12.531377 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524551 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:08:12.531377 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524553 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:08:12.531377 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524555 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:08:12.531377 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524558 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:08:12.531377 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:12.524560 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:08:12.531377 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.524565 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 19:08:12.531377 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.524689 2562 server.go:962] "Client rotation is on, will bootstrap in background" Jul 02 19:08:12.531377 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.526691 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 02 19:08:12.531377 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.528069 2562 server.go:1019] "Starting client certificate rotation" Jul 02 19:08:12.531377 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.528189 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 19:08:12.531377 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.528231 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 19:08:12.531377 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.530930 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 19:08:12.532854 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.532839 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 19:08:12.546605 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.546573 2562 log.go:25] "Validated CRI v1 runtime API" Jul 02 19:08:12.549348 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.549334 2562 log.go:25] "Validated CRI v1 image API" Jul 02 19:08:12.550124 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.550097 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 02 19:08:12.552680 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.552662 2562 fs.go:135] Filesystem UUIDs: map[4b43aa49-d570-4df4-ac68-d388077fb7e0:/dev/nvme0n1p4 68cc83f1-8026-4512-9da9-6e021c204c4b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 02 19:08:12.552753 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.552679 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 02 19:08:12.554023 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.554005 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 19:08:12.559623 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.559507 2562 manager.go:216] Machine: {Timestamp:2026-07-02 19:08:12.558241255 +0000 UTC m=+0.335258833 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107236 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23ec41c8772a6d41fc65428c37ebb8 SystemUUID:ec23ec41-c877-2a6d-41fc-65428c37ebb8 BootID:3d90c995-0bd6-47d0-87f8-91268adbf98e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4a:6a:80:32:0b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4a:6a:80:32:0b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:b0:d0:e3:95:23 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 02 19:08:12.559623 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.559618 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 02 19:08:12.559717 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.559688 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 02 19:08:12.563137 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.563116 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 02 19:08:12.563264 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.563139 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-147.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 02 19:08:12.563841 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.563831 2562 topology_manager.go:138] "Creating topology manager with none policy" Jul 02 19:08:12.563871 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.563843 2562 container_manager_linux.go:309] "Creating device plugin manager" Jul 02 19:08:12.563871 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.563850 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 19:08:12.564548 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.564539 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 19:08:12.565789 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.565779 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 02 19:08:12.568925 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.568914 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 02 19:08:12.569037 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.569028 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 02 19:08:12.569065 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.569059 2562 kubelet.go:486] "Attempting to sync node with API server" Jul 02 19:08:12.569092 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.569071 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 02 19:08:12.569118 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.569095 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 02 19:08:12.569118 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.569115 2562 kubelet.go:398] "Adding apiserver pod source" Jul 02 19:08:12.569186 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.569131 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 02 19:08:12.569611 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.569577 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 19:08:12.569704 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.569629 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 19:08:12.573551 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.573527 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 02 19:08:12.575384 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.575370 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 02 19:08:12.575445 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.575401 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 02 19:08:12.576462 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.576451 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 02 19:08:12.576502 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.576467 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 02 19:08:12.576502 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.576473 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 02 19:08:12.576502 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.576479 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 02 19:08:12.576502 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.576491 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 02 19:08:12.576502 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.576499 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 02 19:08:12.576648 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.576510 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 02 19:08:12.576648 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.576518 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 02 19:08:12.576648 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.576528 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 02 19:08:12.576648 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.576536 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 02 19:08:12.576648 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.576548 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 02 19:08:12.576648 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.576553 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 02 19:08:12.577277 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.577266 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 02 19:08:12.577305 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.577278 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 02 19:08:12.580645 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.580476 2562 server.go:1268] "Started kubelet" Jul 02 19:08:12.580729 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.580569 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 02 19:08:12.580729 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.580563 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 02 19:08:12.580729 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.580725 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 02 19:08:12.581502 ip-10-0-133-147 systemd[1]: Started Kubernetes Kubelet. Jul 02 19:08:12.581944 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.581899 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 02 19:08:12.583365 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.583348 2562 server.go:310] "Adding debug handlers to kubelet server" Jul 02 19:08:12.584499 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.584478 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-147.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 02 19:08:12.584917 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:12.584898 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 02 19:08:12.584968 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:12.584900 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 02 19:08:12.585899 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.585881 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tv6tl" Jul 02 19:08:12.586871 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:12.585920 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-147.ec2.internal.18be8e9989ed1201 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-147.ec2.internal,UID:ip-10-0-133-147.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-147.ec2.internal,},FirstTimestamp:2026-07-02 19:08:12.580450817 +0000 UTC m=+0.357468375,LastTimestamp:2026-07-02 19:08:12.580450817 +0000 UTC m=+0.357468375,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-147.ec2.internal,}" Jul 02 19:08:12.587445 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:12.587418 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 02 19:08:12.587745 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.587733 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 02 19:08:12.587827 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.587808 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 02 19:08:12.587880 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:12.587854 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found" Jul 02 19:08:12.588525 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.588378 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 02 19:08:12.588525 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.588385 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 02 19:08:12.588525 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.588398 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 02 19:08:12.588525 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.588501 2562 reconstruct.go:97] "Volume reconstruction finished" Jul 02 19:08:12.588525 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.588513 2562 reconciler.go:29] "Reconciler: start to sync state" Jul 02 19:08:12.588818 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.588644 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 02 19:08:12.588818 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.588655 2562 factory.go:55] Registering systemd factory Jul 02 19:08:12.588818 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.588664 2562 factory.go:223] Registration of the systemd container factory successfully Jul 02 19:08:12.589041 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.589027 2562 factory.go:153] Registering CRI-O factory Jul 02 19:08:12.589088 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.589044 2562 factory.go:223] Registration of the crio container factory successfully Jul 02 19:08:12.589088 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.589067 2562 factory.go:103] Registering Raw factory Jul 02 19:08:12.589088 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.589080 2562 manager.go:1196] Started watching for new ooms in manager Jul 02 19:08:12.589607 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.589565 2562 manager.go:349] Starting recovery of all containers Jul 02 19:08:12.595880 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.595858 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tv6tl" Jul 02 19:08:12.598684 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:12.598658 2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 02 19:08:12.598788 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:12.598695 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 02 19:08:12.600059 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.600042 2562 manager.go:354] Recovery completed Jul 02 19:08:12.604399 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.604385 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:08:12.606169 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.606152 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:08:12.606238 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.606182 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:08:12.606238 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.606194 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:08:12.606780 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.606765 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 02 19:08:12.606780 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.606776 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:08:12.606912 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.606779 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 02 19:08:12.606912 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.606812 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 02 19:08:12.607420 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.607404 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:08:12.607490 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.607437 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:08:12.607490 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.607453 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:08:12.608046 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.608033 2562 policy_none.go:49] "None policy: Start" Jul 02 19:08:12.608098 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.608050 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 02 19:08:12.608098 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.608061 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 02 19:08:12.609238 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.609227 2562 policy_none.go:47] "Start" Jul 02 19:08:12.646208 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.646189 2562 manager.go:337] "Starting Device Plugin manager" Jul 02 19:08:12.646297 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:12.646257 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 02 19:08:12.646297 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.646270 2562 server.go:87] "Starting device plugin registration server" Jul 02 19:08:12.646491 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.646478 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 02 19:08:12.646540 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.646490 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 02 19:08:12.646608 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.646576 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 02 19:08:12.646876 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.646680 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 02 19:08:12.646876 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.646692 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 02 19:08:12.647076 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:12.647060 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 02 19:08:12.647128 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:12.647099 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-147.ec2.internal\" not found" Jul 02 19:08:12.746977 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.746953 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:08:12.748062 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.747883 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:08:12.748062 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.747914 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:08:12.748062 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.747927 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:08:12.748062 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.747964 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-147.ec2.internal" Jul 02 19:08:12.757032 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.756995 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-147.ec2.internal" Jul 02 19:08:12.757032 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:12.757015 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-147.ec2.internal\": node \"ip-10-0-133-147.ec2.internal\" not found" Jul 02 19:08:12.769184 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.769164 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 02 19:08:12.770473 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.770452 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 02 19:08:12.770563 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.770489 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 02 19:08:12.770563 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.770513 2562 kubelet.go:2448] "Starting kubelet main sync loop" Jul 02 19:08:12.770563 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:12.770551 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 02 19:08:12.772551 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:12.772533 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found" Jul 02 19:08:12.772722 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.772709 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:08:12.871459 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.871429 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-147.ec2.internal"] Jul 02 19:08:12.871583 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.871521 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:08:12.872800 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:12.872785 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found" Jul 02 19:08:12.873458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.873445 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:08:12.873530 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.873475 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:08:12.873530 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.873486 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:08:12.874936 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.874924 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:08:12.875047 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.875032 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal" Jul 02 19:08:12.875084 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.875064 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:08:12.875662 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.875640 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:08:12.875763 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.875672 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:08:12.875763 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.875686 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:08:12.875763 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.875643 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:08:12.875763 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.875713 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:08:12.875763 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.875733 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:08:12.877085 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.877072 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-147.ec2.internal" Jul 02 19:08:12.877131 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.877097 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:08:12.877795 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.877781 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:08:12.877880 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.877810 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:08:12.877880 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.877820 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:08:12.889972 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.889950 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/634e6027337a88ddac8fcf1b6627c64d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal\" (UID: \"634e6027337a88ddac8fcf1b6627c64d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal" Jul 02 19:08:12.890048 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.889975 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d10855b2eb41ac40b2a760c19b9dbc20-config\") pod \"kube-apiserver-proxy-ip-10-0-133-147.ec2.internal\" (UID: \"d10855b2eb41ac40b2a760c19b9dbc20\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-147.ec2.internal" Jul 02 19:08:12.890048 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.889989 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/634e6027337a88ddac8fcf1b6627c64d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal\" (UID: \"634e6027337a88ddac8fcf1b6627c64d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal" Jul 02 19:08:12.902358 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:12.902340 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-147.ec2.internal\" not found" node="ip-10-0-133-147.ec2.internal" Jul 02 19:08:12.906967 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:12.906952 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-147.ec2.internal\" not found" node="ip-10-0-133-147.ec2.internal" Jul 02 19:08:12.973867 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:12.973838 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found" Jul 02 19:08:12.990579 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.990558 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/634e6027337a88ddac8fcf1b6627c64d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal\" (UID: \"634e6027337a88ddac8fcf1b6627c64d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal" Jul 02 19:08:12.990677 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.990586 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/634e6027337a88ddac8fcf1b6627c64d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal\" (UID: \"634e6027337a88ddac8fcf1b6627c64d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal" Jul 02 19:08:12.990677 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.990624 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d10855b2eb41ac40b2a760c19b9dbc20-config\") pod \"kube-apiserver-proxy-ip-10-0-133-147.ec2.internal\" (UID: \"d10855b2eb41ac40b2a760c19b9dbc20\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-147.ec2.internal" Jul 02 19:08:12.990677 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.990643 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/634e6027337a88ddac8fcf1b6627c64d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal\" (UID: \"634e6027337a88ddac8fcf1b6627c64d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal" Jul 02 19:08:12.990677 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.990654 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d10855b2eb41ac40b2a760c19b9dbc20-config\") pod \"kube-apiserver-proxy-ip-10-0-133-147.ec2.internal\" (UID: \"d10855b2eb41ac40b2a760c19b9dbc20\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-147.ec2.internal" Jul 02 19:08:12.990677 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:12.990662 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/634e6027337a88ddac8fcf1b6627c64d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal\" (UID: \"634e6027337a88ddac8fcf1b6627c64d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal" Jul 02 19:08:13.074692 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:13.074643 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found" Jul 02 19:08:13.175135 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:13.175106 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found" Jul 02 19:08:13.205640 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:13.205623 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal" Jul 02 19:08:13.209666 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:13.209584 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-147.ec2.internal" Jul 02 19:08:13.277021 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:13.276988 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found" Jul 02 19:08:13.383586 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:13.383534 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found" Jul 02 19:08:13.484145 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:13.484125 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found" Jul 02 19:08:13.527605 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:13.527563 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 02 19:08:13.528205 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:13.527723 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 19:08:13.584253 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:13.584230 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found" Jul 02 19:08:13.602029 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:13.601999 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 19:03:13 +0000 UTC" deadline="2028-02-19 20:32:29.249951222 +0000 UTC" Jul 02 19:08:13.602029 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:13.602026 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14329h24m15.647929415s" Jul 02 19:08:13.686945 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:13.686888 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found" Jul 02 19:08:13.788055 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:13.788030 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found" Jul 02 19:08:13.888826 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:13.888800 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-147.ec2.internal\" not found" Jul 02 19:08:13.916689 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:13.916638 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10855b2eb41ac40b2a760c19b9dbc20.slice/crio-331079666c2afed6d152cbd411fd72b2f996a1b445fc3d16bee08182ab356e13 WatchSource:0}: Error finding container 331079666c2afed6d152cbd411fd72b2f996a1b445fc3d16bee08182ab356e13: Status 404 returned error can't find the container with id 331079666c2afed6d152cbd411fd72b2f996a1b445fc3d16bee08182ab356e13 Jul 02 19:08:13.916942 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:13.916921 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634e6027337a88ddac8fcf1b6627c64d.slice/crio-f204411b161d9b89335934a69bc8bc98cbe29d3ecf11535f0ef2c0bf09c199eb WatchSource:0}: Error finding container f204411b161d9b89335934a69bc8bc98cbe29d3ecf11535f0ef2c0bf09c199eb: Status 404 returned error can't find the container with id f204411b161d9b89335934a69bc8bc98cbe29d3ecf11535f0ef2c0bf09c199eb Jul 02 19:08:13.924756 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:13.924737 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:08:13.931233 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:13.931217 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:08:13.952947 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:13.952898 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:08:13.989161 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:13.989137 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-147.ec2.internal" Jul 02 19:08:13.999892 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:13.999875 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 19:08:14.001797 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.001784 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal" Jul 02 19:08:14.015211 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.015195 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 19:08:14.143782 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.143764 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:08:14.572411 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.572379 2562 apiserver.go:52] "Watching apiserver" Jul 02 19:08:14.580467 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.580438 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 02 19:08:14.580999 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.580974 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-45nlz","openshift-image-registry/node-ca-xzqgm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal","openshift-multus/multus-additional-cni-plugins-tbph5","openshift-multus/network-metrics-daemon-6z7f2","kube-system/konnectivity-agent-75ftq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq","openshift-multus/multus-d99qz","openshift-network-diagnostics/network-check-target-mhpt4","openshift-network-operator/iptables-alerter-5pm6h","openshift-ovn-kubernetes/ovnkube-node-xm2wv","kube-system/global-pull-secret-syncer-hqszg","kube-system/kube-apiserver-proxy-ip-10-0-133-147.ec2.internal","openshift-cluster-node-tuning-operator/tuned-gnxkm"] Jul 02 19:08:14.582880 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.582862 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.584795 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.584773 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.585764 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.585743 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 02 19:08:14.585926 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.585908 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 02 19:08:14.586009 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.585922 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 02 19:08:14.586009 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.585983 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q25kw\"" Jul 02 19:08:14.587401 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.587359 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 02 19:08:14.587662 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.587622 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xd8wh\"" Jul 02 19:08:14.587876 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.587860 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 02 19:08:14.588310 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.588259 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 02 19:08:14.589039 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.589011 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 02 19:08:14.589247 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.589228 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 02 19:08:14.589503 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.589481 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.589652 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.589604 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.590134 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.590114 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 02 19:08:14.590424 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.590399 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 02 19:08:14.590511 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.590496 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:14.590662 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:14.590561 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6z7f2" podUID="6a167d1c-5370-45bb-96c3-9deb3c9ddb50" Jul 02 19:08:14.592004 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.591984 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:08:14.592476 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.592439 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 02 19:08:14.592571 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.592498 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 02 19:08:14.592838 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.592817 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 02 19:08:14.592929 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.592892 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 02 19:08:14.593019 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.593002 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dt5td\"" Jul 02 19:08:14.593095 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.593078 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-28tn2\"" Jul 02 19:08:14.593196 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.593177 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 02 19:08:14.593250 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.593213 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 02 19:08:14.593861 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.593837 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-75ftq" Jul 02 19:08:14.596002 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.595982 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:14.596096 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:14.596042 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhpt4" podUID="b552dfdb-6037-4463-a58d-59a748531ee1" Jul 02 19:08:14.596378 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.596357 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wnwhj\"" Jul 02 19:08:14.596605 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.596589 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 02 19:08:14.596803 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.596790 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 02 19:08:14.597934 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.597915 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.598017 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.597957 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-ovnkube-config\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.598017 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.597989 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-lib-modules\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.598117 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598017 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7154ce25-e0a8-4d68-8a63-10b774b834cc-system-cni-dir\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.598117 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598042 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7dd39a96-cf82-49eb-a56d-4b0e4cf5b891-sys-fs\") pod \"aws-ebs-csi-driver-node-48svq\" (UID: \"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.598117 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598093 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-systemd-units\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.598272 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598132 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-node-log\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.598272 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598163 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-etc-tuned\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.598272 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598189 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg5ph\" (UniqueName: \"kubernetes.io/projected/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-kube-api-access-qg5ph\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.598272 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598218 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgt2d\" (UniqueName: \"kubernetes.io/projected/7154ce25-e0a8-4d68-8a63-10b774b834cc-kube-api-access-cgt2d\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.598272 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598243 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/270e3b43-4182-4af0-ba88-b327c157291e-agent-certs\") pod \"konnectivity-agent-75ftq\" (UID: \"270e3b43-4182-4af0-ba88-b327c157291e\") " pod="kube-system/konnectivity-agent-75ftq" Jul 02 19:08:14.598525 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598278 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7dd39a96-cf82-49eb-a56d-4b0e4cf5b891-etc-selinux\") pod \"aws-ebs-csi-driver-node-48svq\" (UID: \"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.598525 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598308 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-run-openvswitch\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.598525 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598326 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dvkj\" (UniqueName: \"kubernetes.io/projected/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-kube-api-access-4dvkj\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.598525 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598354 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-etc-sysctl-conf\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.598525 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598386 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-host\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.598525 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598414 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/270e3b43-4182-4af0-ba88-b327c157291e-konnectivity-ca\") pod \"konnectivity-agent-75ftq\" (UID: \"270e3b43-4182-4af0-ba88-b327c157291e\") " pod="kube-system/konnectivity-agent-75ftq" Jul 02 19:08:14.598525 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598437 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5lsv\" (UniqueName: \"kubernetes.io/projected/7dd39a96-cf82-49eb-a56d-4b0e4cf5b891-kube-api-access-c5lsv\") pod \"aws-ebs-csi-driver-node-48svq\" (UID: \"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.598525 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598462 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-log-socket\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.598525 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598498 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.598525 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598523 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-ovn-node-metrics-cert\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.599120 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598542 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-etc-sysconfig\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.599120 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598559 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-etc-kubernetes\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.599120 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598576 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-etc-sysctl-d\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.599120 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598628 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-var-lib-kubelet\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.599120 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598654 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-host-run-netns\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.599120 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598680 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-etc-openvswitch\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.599120 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598705 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-host-run-ovn-kubernetes\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.599120 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598729 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-ovnkube-script-lib\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.599120 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598753 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7154ce25-e0a8-4d68-8a63-10b774b834cc-os-release\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.599120 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598777 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7154ce25-e0a8-4d68-8a63-10b774b834cc-cni-binary-copy\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.599120 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598831 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7154ce25-e0a8-4d68-8a63-10b774b834cc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.599120 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598892 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-host-slash\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.599120 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598947 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-run-systemd\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.599120 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.598984 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-run-ovn\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.599120 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.599014 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-host-cni-bin\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.599120 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.599041 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs\") pod \"network-metrics-daemon-6z7f2\" (UID: \"6a167d1c-5370-45bb-96c3-9deb3c9ddb50\") " pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:14.599883 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.599066 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-host-kubelet\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.599883 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.599094 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7dd39a96-cf82-49eb-a56d-4b0e4cf5b891-socket-dir\") pod \"aws-ebs-csi-driver-node-48svq\" (UID: \"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.599883 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.599122 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7dd39a96-cf82-49eb-a56d-4b0e4cf5b891-device-dir\") pod \"aws-ebs-csi-driver-node-48svq\" (UID: \"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.599883 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.599151 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-var-lib-openvswitch\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.599883 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.599167 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-etc-modprobe-d\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.599883 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.599181 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-sys\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.599883 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.599202 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7154ce25-e0a8-4d68-8a63-10b774b834cc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.599883 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.599235 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-etc-systemd\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.599883 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.599270 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-run\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.599883 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.599306 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-tmp\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.599883 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.599337 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7154ce25-e0a8-4d68-8a63-10b774b834cc-cnibin\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.599883 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.599429 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7154ce25-e0a8-4d68-8a63-10b774b834cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.599883 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.599465 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp5lr\" (UniqueName: \"kubernetes.io/projected/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-kube-api-access-mp5lr\") pod \"network-metrics-daemon-6z7f2\" (UID: \"6a167d1c-5370-45bb-96c3-9deb3c9ddb50\") " pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:14.599883 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.599499 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-env-overrides\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.599883 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.599550 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7dd39a96-cf82-49eb-a56d-4b0e4cf5b891-kubelet-dir\") pod \"aws-ebs-csi-driver-node-48svq\" (UID: \"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.599883 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.599576 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7dd39a96-cf82-49eb-a56d-4b0e4cf5b891-registration-dir\") pod \"aws-ebs-csi-driver-node-48svq\" (UID: \"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.600753 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.599617 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-host-cni-netd\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.600753 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.599985 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xzqgm" Jul 02 19:08:14.600753 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.600500 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 19:08:14.600753 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.600579 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-g7gsx\"" Jul 02 19:08:14.600753 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.600652 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 02 19:08:14.602322 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.602256 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5pm6h" Jul 02 19:08:14.602884 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.602795 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 02 19:08:14.603012 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.602995 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-69wfj\"" Jul 02 19:08:14.603210 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.603166 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 02 19:08:14.604770 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.604449 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 02 19:08:14.604770 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.604455 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-45nlz" Jul 02 19:08:14.604959 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.604914 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 02 19:08:14.605108 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.605088 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:08:14.605238 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.605212 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-q4lpx\"" Jul 02 19:08:14.605305 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.605278 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 02 19:08:14.608738 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.607943 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:14.608738 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:14.608033 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqszg" podUID="448316b6-cd2d-4c73-8ca6-103e4c64f18e" Jul 02 19:08:14.612256 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.612233 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 02 19:08:14.612442 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.612420 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-992l8\"" Jul 02 19:08:14.612547 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.612455 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 02 19:08:14.620064 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.619935 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xvndx" Jul 02 19:08:14.627932 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.627908 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xvndx" Jul 02 19:08:14.689957 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.689886 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 02 19:08:14.699900 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.699874 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp5lr\" (UniqueName: \"kubernetes.io/projected/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-kube-api-access-mp5lr\") pod \"network-metrics-daemon-6z7f2\" (UID: \"6a167d1c-5370-45bb-96c3-9deb3c9ddb50\") " pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:14.699987 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.699913 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-host-kubelet\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.699987 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.699940 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-etc-modprobe-d\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.699987 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.699964 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-sys\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.700179 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.699987 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7154ce25-e0a8-4d68-8a63-10b774b834cc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.700179 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700013 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-etc-systemd\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.700179 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700037 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7154ce25-e0a8-4d68-8a63-10b774b834cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.700179 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700066 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlzhd\" (UniqueName: \"kubernetes.io/projected/51243d8b-5625-444e-bc70-fa68e9b13ecb-kube-api-access-nlzhd\") pod \"node-ca-xzqgm\" (UID: \"51243d8b-5625-444e-bc70-fa68e9b13ecb\") " pod="openshift-image-registry/node-ca-xzqgm" Jul 02 19:08:14.700179 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700093 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-system-cni-dir\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.700179 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700119 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c07acb28-0571-41c0-8a02-55cac407c73d-tmp-dir\") pod \"node-resolver-45nlz\" (UID: \"c07acb28-0571-41c0-8a02-55cac407c73d\") " pod="openshift-dns/node-resolver-45nlz" Jul 02 19:08:14.700179 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700145 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzmxl\" (UniqueName: \"kubernetes.io/projected/c07acb28-0571-41c0-8a02-55cac407c73d-kube-api-access-rzmxl\") pod \"node-resolver-45nlz\" (UID: \"c07acb28-0571-41c0-8a02-55cac407c73d\") " pod="openshift-dns/node-resolver-45nlz" Jul 02 19:08:14.700179 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700172 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7dd39a96-cf82-49eb-a56d-4b0e4cf5b891-registration-dir\") pod \"aws-ebs-csi-driver-node-48svq\" (UID: \"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.700463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700198 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-host-cni-netd\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.700463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700222 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-ovnkube-config\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.700463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700246 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-host-run-k8s-cni-cncf-io\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.700463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700270 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-host-var-lib-cni-multus\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.700463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700295 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-hostroot\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.700463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700324 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-systemd-units\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.700463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700349 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-node-log\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.700463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700405 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-etc-tuned\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.700463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700433 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg5ph\" (UniqueName: \"kubernetes.io/projected/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-kube-api-access-qg5ph\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.700463 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700460 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g85rm\" (UniqueName: \"kubernetes.io/projected/93122604-fc12-4759-a3ab-c12cd59219fa-kube-api-access-g85rm\") pod \"iptables-alerter-5pm6h\" (UID: \"93122604-fc12-4759-a3ab-c12cd59219fa\") " pod="openshift-network-operator/iptables-alerter-5pm6h" Jul 02 19:08:14.700791 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700484 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/448316b6-cd2d-4c73-8ca6-103e4c64f18e-dbus\") pod \"global-pull-secret-syncer-hqszg\" (UID: \"448316b6-cd2d-4c73-8ca6-103e4c64f18e\") " pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:14.700791 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700509 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-run-openvswitch\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.700791 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700550 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dvkj\" (UniqueName: \"kubernetes.io/projected/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-kube-api-access-4dvkj\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.700791 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700588 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-host\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.700791 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700637 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/270e3b43-4182-4af0-ba88-b327c157291e-konnectivity-ca\") pod \"konnectivity-agent-75ftq\" (UID: \"270e3b43-4182-4af0-ba88-b327c157291e\") " pod="kube-system/konnectivity-agent-75ftq" Jul 02 19:08:14.700791 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700697 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-log-socket\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.700791 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700734 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7154ce25-e0a8-4d68-8a63-10b774b834cc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.700791 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700741 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-ovn-node-metrics-cert\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.700791 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700772 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-etc-sysconfig\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.701058 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700819 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-host-kubelet\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.701058 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700833 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-etc-sysconfig\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.701058 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700851 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-etc-kubernetes\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.701058 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700879 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-etc-sysctl-d\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.701058 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700882 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-etc-systemd\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.701058 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700906 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-var-lib-kubelet\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.701058 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700947 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnwvq\" (UniqueName: \"kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq\") pod \"network-check-target-mhpt4\" (UID: \"b552dfdb-6037-4463-a58d-59a748531ee1\") " pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:14.701058 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.700973 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28d07600-1581-4f70-a507-cd1cf2c8b2e2-cni-binary-copy\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.701058 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.701001 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-host-run-netns\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.701058 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.701030 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7154ce25-e0a8-4d68-8a63-10b774b834cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.701058 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.701040 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-ovnkube-script-lib\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.701370 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.701065 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/28d07600-1581-4f70-a507-cd1cf2c8b2e2-multus-daemon-config\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.701370 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.701090 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93122604-fc12-4759-a3ab-c12cd59219fa-host-slash\") pod \"iptables-alerter-5pm6h\" (UID: \"93122604-fc12-4759-a3ab-c12cd59219fa\") " pod="openshift-network-operator/iptables-alerter-5pm6h" Jul 02 19:08:14.701370 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.701195 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7dd39a96-cf82-49eb-a56d-4b0e4cf5b891-registration-dir\") pod \"aws-ebs-csi-driver-node-48svq\" (UID: \"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.701370 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.701232 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-host-cni-netd\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.701370 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.701263 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c07acb28-0571-41c0-8a02-55cac407c73d-hosts-file\") pod \"node-resolver-45nlz\" (UID: \"c07acb28-0571-41c0-8a02-55cac407c73d\") " pod="openshift-dns/node-resolver-45nlz" Jul 02 19:08:14.701370 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.701292 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret\") pod \"global-pull-secret-syncer-hqszg\" (UID: \"448316b6-cd2d-4c73-8ca6-103e4c64f18e\") " pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:14.701370 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.701320 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-run-systemd\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.701370 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.701352 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-host-cni-bin\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.701599 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.701406 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 02 19:08:14.701599 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:14.701520 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:08:14.701599 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:14.701582 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs podName:6a167d1c-5370-45bb-96c3-9deb3c9ddb50 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:15.201561582 +0000 UTC m=+2.978579150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs") pod "network-metrics-daemon-6z7f2" (UID: "6a167d1c-5370-45bb-96c3-9deb3c9ddb50") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:08:14.702475 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.701804 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-systemd-units\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.702475 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.701854 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-host-run-netns\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.702475 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.701898 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-node-log\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.702475 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.701942 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-run-openvswitch\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.702475 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.702046 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-run-systemd\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.702475 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.702113 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-log-socket\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.702475 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.702360 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-ovnkube-config\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.702475 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.702440 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-ovnkube-script-lib\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.702937 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.701424 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs\") pod \"network-metrics-daemon-6z7f2\" (UID: \"6a167d1c-5370-45bb-96c3-9deb3c9ddb50\") " pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:14.702937 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.702511 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/270e3b43-4182-4af0-ba88-b327c157291e-konnectivity-ca\") pod \"konnectivity-agent-75ftq\" (UID: \"270e3b43-4182-4af0-ba88-b327c157291e\") " pod="kube-system/konnectivity-agent-75ftq" Jul 02 19:08:14.702937 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.702583 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-var-lib-kubelet\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.702937 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.702638 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-etc-modprobe-d\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.702937 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.702645 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-host-cni-bin\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.702937 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.702646 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-host\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.702937 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.702674 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-multus-conf-dir\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.702937 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.702698 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-etc-kubernetes\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.702937 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.702586 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-sys\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.702937 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.702709 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-host-run-multus-certs\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.702937 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.702765 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7dd39a96-cf82-49eb-a56d-4b0e4cf5b891-socket-dir\") pod \"aws-ebs-csi-driver-node-48svq\" (UID: \"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.702937 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.702777 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-etc-sysctl-d\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.702937 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.702880 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7dd39a96-cf82-49eb-a56d-4b0e4cf5b891-socket-dir\") pod \"aws-ebs-csi-driver-node-48svq\" (UID: \"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.702937 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.702873 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7dd39a96-cf82-49eb-a56d-4b0e4cf5b891-device-dir\") pod \"aws-ebs-csi-driver-node-48svq\" (UID: \"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.703454 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703094 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-var-lib-openvswitch\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.703454 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703122 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-etc-kubernetes\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.703454 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703149 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-run\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.703454 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703172 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-tmp\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.703454 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703196 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7154ce25-e0a8-4d68-8a63-10b774b834cc-cnibin\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.703454 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703217 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-cnibin\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.703454 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703239 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-host-var-lib-kubelet\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.703454 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703261 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-env-overrides\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.703454 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703284 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7dd39a96-cf82-49eb-a56d-4b0e4cf5b891-kubelet-dir\") pod \"aws-ebs-csi-driver-node-48svq\" (UID: \"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.703454 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703307 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-lib-modules\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.703454 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703333 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7154ce25-e0a8-4d68-8a63-10b774b834cc-system-cni-dir\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.703454 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703358 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/51243d8b-5625-444e-bc70-fa68e9b13ecb-serviceca\") pod \"node-ca-xzqgm\" (UID: \"51243d8b-5625-444e-bc70-fa68e9b13ecb\") " pod="openshift-image-registry/node-ca-xzqgm" Jul 02 19:08:14.703454 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703384 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-multus-cni-dir\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.703454 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703410 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-multus-socket-dir-parent\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.704079 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703473 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7dd39a96-cf82-49eb-a56d-4b0e4cf5b891-sys-fs\") pod \"aws-ebs-csi-driver-node-48svq\" (UID: \"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.704079 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703504 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgt2d\" (UniqueName: \"kubernetes.io/projected/7154ce25-e0a8-4d68-8a63-10b774b834cc-kube-api-access-cgt2d\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.704079 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703531 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/270e3b43-4182-4af0-ba88-b327c157291e-agent-certs\") pod \"konnectivity-agent-75ftq\" (UID: \"270e3b43-4182-4af0-ba88-b327c157291e\") " pod="kube-system/konnectivity-agent-75ftq" Jul 02 19:08:14.704079 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703559 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-host-var-lib-cni-bin\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.704079 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703587 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7dd39a96-cf82-49eb-a56d-4b0e4cf5b891-etc-selinux\") pod \"aws-ebs-csi-driver-node-48svq\" (UID: \"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.704079 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703614 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-etc-sysctl-conf\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.704079 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703642 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5lsv\" (UniqueName: \"kubernetes.io/projected/7dd39a96-cf82-49eb-a56d-4b0e4cf5b891-kube-api-access-c5lsv\") pod \"aws-ebs-csi-driver-node-48svq\" (UID: \"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.704079 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703691 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.704079 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703715 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7dd39a96-cf82-49eb-a56d-4b0e4cf5b891-kubelet-dir\") pod \"aws-ebs-csi-driver-node-48svq\" (UID: \"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.704079 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703723 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7154ce25-e0a8-4d68-8a63-10b774b834cc-os-release\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.704079 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.702940 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7dd39a96-cf82-49eb-a56d-4b0e4cf5b891-device-dir\") pod \"aws-ebs-csi-driver-node-48svq\" (UID: \"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.704079 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703787 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-var-lib-openvswitch\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.704079 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703793 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7154ce25-e0a8-4d68-8a63-10b774b834cc-os-release\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.704079 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703829 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7154ce25-e0a8-4d68-8a63-10b774b834cc-cni-binary-copy\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.704079 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703871 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93122604-fc12-4759-a3ab-c12cd59219fa-iptables-alerter-script\") pod \"iptables-alerter-5pm6h\" (UID: \"93122604-fc12-4759-a3ab-c12cd59219fa\") " pod="openshift-network-operator/iptables-alerter-5pm6h" Jul 02 19:08:14.704079 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703898 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-etc-openvswitch\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.705098 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703924 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-host-run-ovn-kubernetes\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.705098 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703954 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7154ce25-e0a8-4d68-8a63-10b774b834cc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.705098 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703984 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-os-release\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.705098 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.704010 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-host-slash\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.705098 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.704036 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-run-ovn\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.705098 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.704064 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51243d8b-5625-444e-bc70-fa68e9b13ecb-host\") pod \"node-ca-xzqgm\" (UID: \"51243d8b-5625-444e-bc70-fa68e9b13ecb\") " pod="openshift-image-registry/node-ca-xzqgm" Jul 02 19:08:14.705098 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.704089 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-host-run-netns\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.705098 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.704116 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4wph\" (UniqueName: \"kubernetes.io/projected/28d07600-1581-4f70-a507-cd1cf2c8b2e2-kube-api-access-b4wph\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.705098 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.704142 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/448316b6-cd2d-4c73-8ca6-103e4c64f18e-kubelet-config\") pod \"global-pull-secret-syncer-hqszg\" (UID: \"448316b6-cd2d-4c73-8ca6-103e4c64f18e\") " pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:14.705098 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.704223 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.705098 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.704322 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-lib-modules\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.705098 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.704357 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7154ce25-e0a8-4d68-8a63-10b774b834cc-system-cni-dir\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.705098 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.704412 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7dd39a96-cf82-49eb-a56d-4b0e4cf5b891-sys-fs\") pod \"aws-ebs-csi-driver-node-48svq\" (UID: \"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.705098 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.704617 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7154ce25-e0a8-4d68-8a63-10b774b834cc-cnibin\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.705098 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.704706 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7154ce25-e0a8-4d68-8a63-10b774b834cc-cni-binary-copy\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.705098 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.703841 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-run\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.705098 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.704800 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-etc-openvswitch\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.706063 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.704843 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-host-run-ovn-kubernetes\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.706063 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.705110 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-env-overrides\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.706063 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.705271 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7dd39a96-cf82-49eb-a56d-4b0e4cf5b891-etc-selinux\") pod \"aws-ebs-csi-driver-node-48svq\" (UID: \"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.706063 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.705397 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7154ce25-e0a8-4d68-8a63-10b774b834cc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.706063 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.705474 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-etc-sysctl-conf\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.706063 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.705478 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-host-slash\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.706063 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.705513 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-run-ovn\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.706063 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.705818 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-ovn-node-metrics-cert\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.706748 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.706556 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-etc-tuned\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.706748 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.706715 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-tmp\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.708399 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.708377 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/270e3b43-4182-4af0-ba88-b327c157291e-agent-certs\") pod \"konnectivity-agent-75ftq\" (UID: \"270e3b43-4182-4af0-ba88-b327c157291e\") " pod="kube-system/konnectivity-agent-75ftq" Jul 02 19:08:14.713687 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.713646 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp5lr\" (UniqueName: \"kubernetes.io/projected/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-kube-api-access-mp5lr\") pod \"network-metrics-daemon-6z7f2\" (UID: \"6a167d1c-5370-45bb-96c3-9deb3c9ddb50\") " pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:14.715244 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.715190 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg5ph\" (UniqueName: \"kubernetes.io/projected/3c8d06c3-95ff-4490-9fdf-14c5570a0c52-kube-api-access-qg5ph\") pod \"tuned-gnxkm\" (UID: \"3c8d06c3-95ff-4490-9fdf-14c5570a0c52\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.715851 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.715832 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dvkj\" (UniqueName: \"kubernetes.io/projected/f2b9ac49-b05c-4d63-95c2-14d0e77bb163-kube-api-access-4dvkj\") pod \"ovnkube-node-xm2wv\" (UID: \"f2b9ac49-b05c-4d63-95c2-14d0e77bb163\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.717398 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.717377 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgt2d\" (UniqueName: \"kubernetes.io/projected/7154ce25-e0a8-4d68-8a63-10b774b834cc-kube-api-access-cgt2d\") pod \"multus-additional-cni-plugins-tbph5\" (UID: \"7154ce25-e0a8-4d68-8a63-10b774b834cc\") " pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.720187 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.720143 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5lsv\" (UniqueName: \"kubernetes.io/projected/7dd39a96-cf82-49eb-a56d-4b0e4cf5b891-kube-api-access-c5lsv\") pod \"aws-ebs-csi-driver-node-48svq\" (UID: \"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.778994 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.778917 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-147.ec2.internal" event={"ID":"d10855b2eb41ac40b2a760c19b9dbc20","Type":"ContainerStarted","Data":"331079666c2afed6d152cbd411fd72b2f996a1b445fc3d16bee08182ab356e13"} Jul 02 19:08:14.781218 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.781156 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal" event={"ID":"634e6027337a88ddac8fcf1b6627c64d","Type":"ContainerStarted","Data":"f204411b161d9b89335934a69bc8bc98cbe29d3ecf11535f0ef2c0bf09c199eb"} Jul 02 19:08:14.783709 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.783493 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:08:14.804461 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804433 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/51243d8b-5625-444e-bc70-fa68e9b13ecb-serviceca\") pod \"node-ca-xzqgm\" (UID: \"51243d8b-5625-444e-bc70-fa68e9b13ecb\") " pod="openshift-image-registry/node-ca-xzqgm" Jul 02 19:08:14.804584 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804477 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-multus-cni-dir\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.804584 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804506 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-multus-socket-dir-parent\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.804584 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804535 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-host-var-lib-cni-bin\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.804584 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804566 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93122604-fc12-4759-a3ab-c12cd59219fa-iptables-alerter-script\") pod \"iptables-alerter-5pm6h\" (UID: \"93122604-fc12-4759-a3ab-c12cd59219fa\") " pod="openshift-network-operator/iptables-alerter-5pm6h" Jul 02 19:08:14.804835 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804594 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-os-release\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.804835 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804641 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51243d8b-5625-444e-bc70-fa68e9b13ecb-host\") pod \"node-ca-xzqgm\" (UID: \"51243d8b-5625-444e-bc70-fa68e9b13ecb\") " pod="openshift-image-registry/node-ca-xzqgm" Jul 02 19:08:14.804835 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804668 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-host-run-netns\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.804835 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804691 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4wph\" (UniqueName: \"kubernetes.io/projected/28d07600-1581-4f70-a507-cd1cf2c8b2e2-kube-api-access-b4wph\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.804835 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804715 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/448316b6-cd2d-4c73-8ca6-103e4c64f18e-kubelet-config\") pod \"global-pull-secret-syncer-hqszg\" (UID: \"448316b6-cd2d-4c73-8ca6-103e4c64f18e\") " pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:14.804835 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804735 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlzhd\" (UniqueName: \"kubernetes.io/projected/51243d8b-5625-444e-bc70-fa68e9b13ecb-kube-api-access-nlzhd\") pod \"node-ca-xzqgm\" (UID: \"51243d8b-5625-444e-bc70-fa68e9b13ecb\") " pod="openshift-image-registry/node-ca-xzqgm" Jul 02 19:08:14.804835 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804750 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-system-cni-dir\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.804835 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804769 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c07acb28-0571-41c0-8a02-55cac407c73d-tmp-dir\") pod \"node-resolver-45nlz\" (UID: \"c07acb28-0571-41c0-8a02-55cac407c73d\") " pod="openshift-dns/node-resolver-45nlz" Jul 02 19:08:14.804835 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804794 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzmxl\" (UniqueName: \"kubernetes.io/projected/c07acb28-0571-41c0-8a02-55cac407c73d-kube-api-access-rzmxl\") pod \"node-resolver-45nlz\" (UID: \"c07acb28-0571-41c0-8a02-55cac407c73d\") " pod="openshift-dns/node-resolver-45nlz" Jul 02 19:08:14.804835 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804822 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-host-run-k8s-cni-cncf-io\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.804835 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804839 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-host-var-lib-cni-multus\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.805458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804864 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-hostroot\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.805458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804883 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g85rm\" (UniqueName: \"kubernetes.io/projected/93122604-fc12-4759-a3ab-c12cd59219fa-kube-api-access-g85rm\") pod \"iptables-alerter-5pm6h\" (UID: \"93122604-fc12-4759-a3ab-c12cd59219fa\") " pod="openshift-network-operator/iptables-alerter-5pm6h" Jul 02 19:08:14.805458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804902 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/448316b6-cd2d-4c73-8ca6-103e4c64f18e-dbus\") pod \"global-pull-secret-syncer-hqszg\" (UID: \"448316b6-cd2d-4c73-8ca6-103e4c64f18e\") " pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:14.805458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804931 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnwvq\" (UniqueName: \"kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq\") pod \"network-check-target-mhpt4\" (UID: \"b552dfdb-6037-4463-a58d-59a748531ee1\") " pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:14.805458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804947 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28d07600-1581-4f70-a507-cd1cf2c8b2e2-cni-binary-copy\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.805458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804965 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/28d07600-1581-4f70-a507-cd1cf2c8b2e2-multus-daemon-config\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.805458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804980 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93122604-fc12-4759-a3ab-c12cd59219fa-host-slash\") pod \"iptables-alerter-5pm6h\" (UID: \"93122604-fc12-4759-a3ab-c12cd59219fa\") " pod="openshift-network-operator/iptables-alerter-5pm6h" Jul 02 19:08:14.805458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.804981 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/51243d8b-5625-444e-bc70-fa68e9b13ecb-serviceca\") pod \"node-ca-xzqgm\" (UID: \"51243d8b-5625-444e-bc70-fa68e9b13ecb\") " pod="openshift-image-registry/node-ca-xzqgm" Jul 02 19:08:14.805458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.805004 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c07acb28-0571-41c0-8a02-55cac407c73d-hosts-file\") pod \"node-resolver-45nlz\" (UID: \"c07acb28-0571-41c0-8a02-55cac407c73d\") " pod="openshift-dns/node-resolver-45nlz" Jul 02 19:08:14.805458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.805021 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret\") pod \"global-pull-secret-syncer-hqszg\" (UID: \"448316b6-cd2d-4c73-8ca6-103e4c64f18e\") " pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:14.805458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.805046 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-multus-conf-dir\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.805458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.805061 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-system-cni-dir\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.805458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.805067 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-host-run-multus-certs\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.805458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.805097 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-etc-kubernetes\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.805458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.805114 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-cnibin\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.805458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.805121 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-multus-cni-dir\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.805458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.805138 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-host-var-lib-kubelet\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.805458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.805173 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-multus-socket-dir-parent\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.806450 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.805216 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-host-var-lib-cni-bin\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.806450 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.805255 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-host-var-lib-kubelet\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.806450 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.805418 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c07acb28-0571-41c0-8a02-55cac407c73d-tmp-dir\") pod \"node-resolver-45nlz\" (UID: \"c07acb28-0571-41c0-8a02-55cac407c73d\") " pod="openshift-dns/node-resolver-45nlz" Jul 02 19:08:14.806450 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.805648 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-host-run-k8s-cni-cncf-io\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.806450 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.805711 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-host-var-lib-cni-multus\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.806450 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.805745 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-hostroot\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.806450 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.805779 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93122604-fc12-4759-a3ab-c12cd59219fa-iptables-alerter-script\") pod \"iptables-alerter-5pm6h\" (UID: \"93122604-fc12-4759-a3ab-c12cd59219fa\") " pod="openshift-network-operator/iptables-alerter-5pm6h" Jul 02 19:08:14.806450 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.805805 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-host-run-netns\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.806450 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.805863 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/448316b6-cd2d-4c73-8ca6-103e4c64f18e-dbus\") pod \"global-pull-secret-syncer-hqszg\" (UID: \"448316b6-cd2d-4c73-8ca6-103e4c64f18e\") " pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:14.806450 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.805975 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-os-release\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.806450 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.806018 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51243d8b-5625-444e-bc70-fa68e9b13ecb-host\") pod \"node-ca-xzqgm\" (UID: \"51243d8b-5625-444e-bc70-fa68e9b13ecb\") " pod="openshift-image-registry/node-ca-xzqgm" Jul 02 19:08:14.806450 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.806076 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/448316b6-cd2d-4c73-8ca6-103e4c64f18e-kubelet-config\") pod \"global-pull-secret-syncer-hqszg\" (UID: \"448316b6-cd2d-4c73-8ca6-103e4c64f18e\") " pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:14.806450 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:14.806152 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:08:14.806450 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:14.806205 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret podName:448316b6-cd2d-4c73-8ca6-103e4c64f18e nodeName:}" failed. No retries permitted until 2026-07-02 19:08:15.306186218 +0000 UTC m=+3.083203781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret") pod "global-pull-secret-syncer-hqszg" (UID: "448316b6-cd2d-4c73-8ca6-103e4c64f18e") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:08:14.806450 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.806417 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28d07600-1581-4f70-a507-cd1cf2c8b2e2-cni-binary-copy\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.807310 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.806468 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-multus-conf-dir\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.807310 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.806498 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-host-run-multus-certs\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.807310 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.806528 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-etc-kubernetes\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.807310 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.806531 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/28d07600-1581-4f70-a507-cd1cf2c8b2e2-multus-daemon-config\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.807310 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.806567 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28d07600-1581-4f70-a507-cd1cf2c8b2e2-cnibin\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.807310 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.806602 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93122604-fc12-4759-a3ab-c12cd59219fa-host-slash\") pod \"iptables-alerter-5pm6h\" (UID: \"93122604-fc12-4759-a3ab-c12cd59219fa\") " pod="openshift-network-operator/iptables-alerter-5pm6h" Jul 02 19:08:14.807310 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.806623 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c07acb28-0571-41c0-8a02-55cac407c73d-hosts-file\") pod \"node-resolver-45nlz\" (UID: \"c07acb28-0571-41c0-8a02-55cac407c73d\") " pod="openshift-dns/node-resolver-45nlz" Jul 02 19:08:14.819123 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:14.819099 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:08:14.819228 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:14.819129 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:08:14.819228 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:14.819144 2562 projected.go:196] Error preparing data for projected volume kube-api-access-hnwvq for pod openshift-network-diagnostics/network-check-target-mhpt4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:08:14.819228 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:14.819213 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq podName:b552dfdb-6037-4463-a58d-59a748531ee1 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:15.319195519 +0000 UTC m=+3.096213084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hnwvq" (UniqueName: "kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq") pod "network-check-target-mhpt4" (UID: "b552dfdb-6037-4463-a58d-59a748531ee1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:08:14.822568 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.822507 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4wph\" (UniqueName: \"kubernetes.io/projected/28d07600-1581-4f70-a507-cd1cf2c8b2e2-kube-api-access-b4wph\") pod \"multus-d99qz\" (UID: \"28d07600-1581-4f70-a507-cd1cf2c8b2e2\") " pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.823256 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.823236 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g85rm\" (UniqueName: \"kubernetes.io/projected/93122604-fc12-4759-a3ab-c12cd59219fa-kube-api-access-g85rm\") pod \"iptables-alerter-5pm6h\" (UID: \"93122604-fc12-4759-a3ab-c12cd59219fa\") " pod="openshift-network-operator/iptables-alerter-5pm6h" Jul 02 19:08:14.823850 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.823803 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzmxl\" (UniqueName: \"kubernetes.io/projected/c07acb28-0571-41c0-8a02-55cac407c73d-kube-api-access-rzmxl\") pod \"node-resolver-45nlz\" (UID: \"c07acb28-0571-41c0-8a02-55cac407c73d\") " pod="openshift-dns/node-resolver-45nlz" Jul 02 19:08:14.826288 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.826264 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlzhd\" (UniqueName: \"kubernetes.io/projected/51243d8b-5625-444e-bc70-fa68e9b13ecb-kube-api-access-nlzhd\") pod \"node-ca-xzqgm\" (UID: \"51243d8b-5625-444e-bc70-fa68e9b13ecb\") " pod="openshift-image-registry/node-ca-xzqgm" Jul 02 19:08:14.897819 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.897794 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" Jul 02 19:08:14.907044 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:14.906866 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd39a96_cf82_49eb_a56d_4b0e4cf5b891.slice/crio-bc440913b43c8210384fe686a462a7b0e99fd55e850b2ada7d5537c24fcf5c6e WatchSource:0}: Error finding container bc440913b43c8210384fe686a462a7b0e99fd55e850b2ada7d5537c24fcf5c6e: Status 404 returned error can't find the container with id bc440913b43c8210384fe686a462a7b0e99fd55e850b2ada7d5537c24fcf5c6e Jul 02 19:08:14.908324 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.908304 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:14.919300 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.919112 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d99qz" Jul 02 19:08:14.919529 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:14.919501 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b9ac49_b05c_4d63_95c2_14d0e77bb163.slice/crio-db7e32862e37b18feedfca079063da09c64ebc269fcc5bd9ca47024b26e6dcb5 WatchSource:0}: Error finding container db7e32862e37b18feedfca079063da09c64ebc269fcc5bd9ca47024b26e6dcb5: Status 404 returned error can't find the container with id db7e32862e37b18feedfca079063da09c64ebc269fcc5bd9ca47024b26e6dcb5 Jul 02 19:08:14.925357 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.925246 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" Jul 02 19:08:14.928358 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:14.928301 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d07600_1581_4f70_a507_cd1cf2c8b2e2.slice/crio-7bf45566ccc285d57b8930c6da1296e6dce5b8516e923112591ec4d1a0c3c985 WatchSource:0}: Error finding container 7bf45566ccc285d57b8930c6da1296e6dce5b8516e923112591ec4d1a0c3c985: Status 404 returned error can't find the container with id 7bf45566ccc285d57b8930c6da1296e6dce5b8516e923112591ec4d1a0c3c985 Jul 02 19:08:14.933840 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.933821 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tbph5" Jul 02 19:08:14.942052 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.942022 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-75ftq" Jul 02 19:08:14.950013 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.949994 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xzqgm" Jul 02 19:08:14.960180 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.960158 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5pm6h" Jul 02 19:08:14.965796 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:14.965778 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-45nlz" Jul 02 19:08:15.209550 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:15.209452 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs\") pod \"network-metrics-daemon-6z7f2\" (UID: \"6a167d1c-5370-45bb-96c3-9deb3c9ddb50\") " pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:15.209716 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:15.209615 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:08:15.209784 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:15.209718 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs podName:6a167d1c-5370-45bb-96c3-9deb3c9ddb50 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:16.209693393 +0000 UTC m=+3.986710961 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs") pod "network-metrics-daemon-6z7f2" (UID: "6a167d1c-5370-45bb-96c3-9deb3c9ddb50") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:08:15.310532 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:15.310504 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret\") pod \"global-pull-secret-syncer-hqszg\" (UID: \"448316b6-cd2d-4c73-8ca6-103e4c64f18e\") " pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:15.310729 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:15.310662 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:08:15.310792 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:15.310729 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret podName:448316b6-cd2d-4c73-8ca6-103e4c64f18e nodeName:}" failed. No retries permitted until 2026-07-02 19:08:16.310711453 +0000 UTC m=+4.087729004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret") pod "global-pull-secret-syncer-hqszg" (UID: "448316b6-cd2d-4c73-8ca6-103e4c64f18e") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:08:15.411752 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:15.411717 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnwvq\" (UniqueName: \"kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq\") pod \"network-check-target-mhpt4\" (UID: \"b552dfdb-6037-4463-a58d-59a748531ee1\") " pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:15.411894 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:15.411873 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:08:15.411894 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:15.411893 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:08:15.411974 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:15.411903 2562 projected.go:196] Error preparing data for projected volume kube-api-access-hnwvq for pod openshift-network-diagnostics/network-check-target-mhpt4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:08:15.411974 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:15.411960 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq podName:b552dfdb-6037-4463-a58d-59a748531ee1 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:16.411944373 +0000 UTC m=+4.188961925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hnwvq" (UniqueName: "kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq") pod "network-check-target-mhpt4" (UID: "b552dfdb-6037-4463-a58d-59a748531ee1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:08:15.630650 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:15.630489 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 19:03:15 +0000 UTC" deadline="2028-02-22 19:17:28.987198058 +0000 UTC" Jul 02 19:08:15.630650 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:15.630528 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14400h9m13.356673975s" Jul 02 19:08:15.772139 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:15.771627 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:15.772139 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:15.771761 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhpt4" podUID="b552dfdb-6037-4463-a58d-59a748531ee1" Jul 02 19:08:15.787806 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:15.787730 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d99qz" event={"ID":"28d07600-1581-4f70-a507-cd1cf2c8b2e2","Type":"ContainerStarted","Data":"7bf45566ccc285d57b8930c6da1296e6dce5b8516e923112591ec4d1a0c3c985"} Jul 02 19:08:15.790355 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:15.790316 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" event={"ID":"f2b9ac49-b05c-4d63-95c2-14d0e77bb163","Type":"ContainerStarted","Data":"db7e32862e37b18feedfca079063da09c64ebc269fcc5bd9ca47024b26e6dcb5"} Jul 02 19:08:15.791802 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:15.791770 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" event={"ID":"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891","Type":"ContainerStarted","Data":"bc440913b43c8210384fe686a462a7b0e99fd55e850b2ada7d5537c24fcf5c6e"} Jul 02 19:08:16.218994 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:16.218964 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs\") pod \"network-metrics-daemon-6z7f2\" (UID: \"6a167d1c-5370-45bb-96c3-9deb3c9ddb50\") " pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:16.219172 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:16.219155 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:08:16.219272 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:16.219225 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs podName:6a167d1c-5370-45bb-96c3-9deb3c9ddb50 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:18.219204841 +0000 UTC m=+5.996222609 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs") pod "network-metrics-daemon-6z7f2" (UID: "6a167d1c-5370-45bb-96c3-9deb3c9ddb50") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:08:16.243617 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:16.243546 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07acb28_0571_41c0_8a02_55cac407c73d.slice/crio-9245a005b62bf8edd654f607c29707c618a75a3cc089148617b9937ae422a7e9 WatchSource:0}: Error finding container 9245a005b62bf8edd654f607c29707c618a75a3cc089148617b9937ae422a7e9: Status 404 returned error can't find the container with id 9245a005b62bf8edd654f607c29707c618a75a3cc089148617b9937ae422a7e9 Jul 02 19:08:16.246022 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:16.245993 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93122604_fc12_4759_a3ab_c12cd59219fa.slice/crio-dd1f29c95edf943a4daee67630c5641e4da4b886f1e96179b961608ea3f65f2f WatchSource:0}: Error finding container dd1f29c95edf943a4daee67630c5641e4da4b886f1e96179b961608ea3f65f2f: Status 404 returned error can't find the container with id dd1f29c95edf943a4daee67630c5641e4da4b886f1e96179b961608ea3f65f2f Jul 02 19:08:16.249215 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:16.249196 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270e3b43_4182_4af0_ba88_b327c157291e.slice/crio-077a12498a96c0b3682171275001881d7adf3d3fef51112ae5d436e4a261ab33 WatchSource:0}: Error finding container 077a12498a96c0b3682171275001881d7adf3d3fef51112ae5d436e4a261ab33: Status 404 returned error can't find the container with id 077a12498a96c0b3682171275001881d7adf3d3fef51112ae5d436e4a261ab33 Jul 02 19:08:16.251273 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:16.251249 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51243d8b_5625_444e_bc70_fa68e9b13ecb.slice/crio-df320aa6d615e18569588a725ef17e9f16e81b8480c8a2218bc70c9291072539 WatchSource:0}: Error finding container df320aa6d615e18569588a725ef17e9f16e81b8480c8a2218bc70c9291072539: Status 404 returned error can't find the container with id df320aa6d615e18569588a725ef17e9f16e81b8480c8a2218bc70c9291072539 Jul 02 19:08:16.255541 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:16.255513 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7154ce25_e0a8_4d68_8a63_10b774b834cc.slice/crio-38881b7a9955b934b1d6c8a68cde8d4ab203db01df8c95955827c495dc517ba5 WatchSource:0}: Error finding container 38881b7a9955b934b1d6c8a68cde8d4ab203db01df8c95955827c495dc517ba5: Status 404 returned error can't find the container with id 38881b7a9955b934b1d6c8a68cde8d4ab203db01df8c95955827c495dc517ba5 Jul 02 19:08:16.319491 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:16.319459 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret\") pod \"global-pull-secret-syncer-hqszg\" (UID: \"448316b6-cd2d-4c73-8ca6-103e4c64f18e\") " pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:16.319619 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:16.319589 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:08:16.319718 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:16.319672 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret podName:448316b6-cd2d-4c73-8ca6-103e4c64f18e nodeName:}" failed. No retries permitted until 2026-07-02 19:08:18.319654338 +0000 UTC m=+6.096671888 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret") pod "global-pull-secret-syncer-hqszg" (UID: "448316b6-cd2d-4c73-8ca6-103e4c64f18e") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:08:16.420196 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:16.420158 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnwvq\" (UniqueName: \"kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq\") pod \"network-check-target-mhpt4\" (UID: \"b552dfdb-6037-4463-a58d-59a748531ee1\") " pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:16.420391 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:16.420367 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:08:16.420451 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:16.420402 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:08:16.420451 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:16.420418 2562 projected.go:196] Error preparing data for projected volume kube-api-access-hnwvq for pod openshift-network-diagnostics/network-check-target-mhpt4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:08:16.420582 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:16.420481 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq podName:b552dfdb-6037-4463-a58d-59a748531ee1 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:18.420460709 +0000 UTC m=+6.197478259 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hnwvq" (UniqueName: "kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq") pod "network-check-target-mhpt4" (UID: "b552dfdb-6037-4463-a58d-59a748531ee1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:08:16.630720 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:16.630653 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 19:03:15 +0000 UTC" deadline="2028-04-02 08:53:47.97907442 +0000 UTC" Jul 02 19:08:16.630720 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:16.630715 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15349h45m31.348364819s" Jul 02 19:08:16.779705 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:16.772961 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:16.779705 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:16.772972 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:16.779705 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:16.773172 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqszg" podUID="448316b6-cd2d-4c73-8ca6-103e4c64f18e" Jul 02 19:08:16.779705 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:16.775583 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6z7f2" podUID="6a167d1c-5370-45bb-96c3-9deb3c9ddb50" Jul 02 19:08:16.799319 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:16.799203 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5pm6h" event={"ID":"93122604-fc12-4759-a3ab-c12cd59219fa","Type":"ContainerStarted","Data":"dd1f29c95edf943a4daee67630c5641e4da4b886f1e96179b961608ea3f65f2f"} Jul 02 19:08:16.805618 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:16.805538 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-75ftq" event={"ID":"270e3b43-4182-4af0-ba88-b327c157291e","Type":"ContainerStarted","Data":"077a12498a96c0b3682171275001881d7adf3d3fef51112ae5d436e4a261ab33"} Jul 02 19:08:16.807563 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:16.807479 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-45nlz" event={"ID":"c07acb28-0571-41c0-8a02-55cac407c73d","Type":"ContainerStarted","Data":"9245a005b62bf8edd654f607c29707c618a75a3cc089148617b9937ae422a7e9"} Jul 02 19:08:16.814701 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:16.814665 2562 generic.go:356] "Generic (PLEG): container finished" podID="634e6027337a88ddac8fcf1b6627c64d" containerID="2f406ffa749d080e040954cef7a4f02b246f982050aefc70bd4796127679117c" exitCode=0 Jul 02 19:08:16.814812 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:16.814751 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal" event={"ID":"634e6027337a88ddac8fcf1b6627c64d","Type":"ContainerDied","Data":"2f406ffa749d080e040954cef7a4f02b246f982050aefc70bd4796127679117c"} Jul 02 19:08:16.818530 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:16.818303 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-147.ec2.internal" event={"ID":"d10855b2eb41ac40b2a760c19b9dbc20","Type":"ContainerStarted","Data":"680e74d40d7d9939918ec032b83353ff01f6cb253cd6730b878df812e6c6fe2e"} Jul 02 19:08:16.820590 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:16.820514 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xzqgm" event={"ID":"51243d8b-5625-444e-bc70-fa68e9b13ecb","Type":"ContainerStarted","Data":"df320aa6d615e18569588a725ef17e9f16e81b8480c8a2218bc70c9291072539"} Jul 02 19:08:16.824245 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:16.823855 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" event={"ID":"3c8d06c3-95ff-4490-9fdf-14c5570a0c52","Type":"ContainerStarted","Data":"c556a071def8015092dae80e224fa3e91ce9f18beddd187cf1e067c283a47d8d"} Jul 02 19:08:16.836173 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:16.836107 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbph5" event={"ID":"7154ce25-e0a8-4d68-8a63-10b774b834cc","Type":"ContainerStarted","Data":"38881b7a9955b934b1d6c8a68cde8d4ab203db01df8c95955827c495dc517ba5"} Jul 02 19:08:17.771290 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:17.770762 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:17.771290 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:17.770900 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhpt4" podUID="b552dfdb-6037-4463-a58d-59a748531ee1" Jul 02 19:08:18.234480 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:18.234391 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs\") pod \"network-metrics-daemon-6z7f2\" (UID: \"6a167d1c-5370-45bb-96c3-9deb3c9ddb50\") " pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:18.234644 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:18.234539 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:08:18.234644 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:18.234604 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs podName:6a167d1c-5370-45bb-96c3-9deb3c9ddb50 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:22.234584788 +0000 UTC m=+10.011602353 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs") pod "network-metrics-daemon-6z7f2" (UID: "6a167d1c-5370-45bb-96c3-9deb3c9ddb50") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:08:18.335023 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:18.334987 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret\") pod \"global-pull-secret-syncer-hqszg\" (UID: \"448316b6-cd2d-4c73-8ca6-103e4c64f18e\") " pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:18.335219 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:18.335122 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:08:18.335219 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:18.335186 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret podName:448316b6-cd2d-4c73-8ca6-103e4c64f18e nodeName:}" failed. No retries permitted until 2026-07-02 19:08:22.335166526 +0000 UTC m=+10.112184076 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret") pod "global-pull-secret-syncer-hqszg" (UID: "448316b6-cd2d-4c73-8ca6-103e4c64f18e") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:08:18.436704 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:18.436006 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnwvq\" (UniqueName: \"kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq\") pod \"network-check-target-mhpt4\" (UID: \"b552dfdb-6037-4463-a58d-59a748531ee1\") " pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:18.436704 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:18.436194 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:08:18.436704 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:18.436213 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:08:18.436704 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:18.436226 2562 projected.go:196] Error preparing data for projected volume kube-api-access-hnwvq for pod openshift-network-diagnostics/network-check-target-mhpt4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:08:18.436704 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:18.436282 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq podName:b552dfdb-6037-4463-a58d-59a748531ee1 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:22.436263841 +0000 UTC m=+10.213281393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hnwvq" (UniqueName: "kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq") pod "network-check-target-mhpt4" (UID: "b552dfdb-6037-4463-a58d-59a748531ee1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:08:18.774019 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:18.773984 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:18.774693 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:18.774129 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqszg" podUID="448316b6-cd2d-4c73-8ca6-103e4c64f18e" Jul 02 19:08:18.774693 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:18.774547 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:18.774886 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:18.774814 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6z7f2" podUID="6a167d1c-5370-45bb-96c3-9deb3c9ddb50" Jul 02 19:08:19.770838 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:19.770806 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:19.771037 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:19.770931 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhpt4" podUID="b552dfdb-6037-4463-a58d-59a748531ee1" Jul 02 19:08:20.771785 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:20.771747 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:20.772273 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:20.771909 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqszg" podUID="448316b6-cd2d-4c73-8ca6-103e4c64f18e" Jul 02 19:08:20.772273 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:20.771934 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:20.772273 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:20.772105 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6z7f2" podUID="6a167d1c-5370-45bb-96c3-9deb3c9ddb50" Jul 02 19:08:21.771110 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:21.771065 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:21.771312 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:21.771201 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhpt4" podUID="b552dfdb-6037-4463-a58d-59a748531ee1" Jul 02 19:08:22.267895 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:22.267808 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs\") pod \"network-metrics-daemon-6z7f2\" (UID: \"6a167d1c-5370-45bb-96c3-9deb3c9ddb50\") " pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:22.268326 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:22.267958 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:08:22.268326 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:22.268035 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs podName:6a167d1c-5370-45bb-96c3-9deb3c9ddb50 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:30.268013424 +0000 UTC m=+18.045030981 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs") pod "network-metrics-daemon-6z7f2" (UID: "6a167d1c-5370-45bb-96c3-9deb3c9ddb50") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:08:22.369181 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:22.369137 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret\") pod \"global-pull-secret-syncer-hqszg\" (UID: \"448316b6-cd2d-4c73-8ca6-103e4c64f18e\") " pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:22.369356 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:22.369297 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:08:22.369356 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:22.369353 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret podName:448316b6-cd2d-4c73-8ca6-103e4c64f18e nodeName:}" failed. No retries permitted until 2026-07-02 19:08:30.369332984 +0000 UTC m=+18.146350529 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret") pod "global-pull-secret-syncer-hqszg" (UID: "448316b6-cd2d-4c73-8ca6-103e4c64f18e") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:08:22.470397 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:22.470357 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnwvq\" (UniqueName: \"kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq\") pod \"network-check-target-mhpt4\" (UID: \"b552dfdb-6037-4463-a58d-59a748531ee1\") " pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:22.470554 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:22.470520 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:08:22.470554 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:22.470541 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:08:22.470554 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:22.470554 2562 projected.go:196] Error preparing data for projected volume kube-api-access-hnwvq for pod openshift-network-diagnostics/network-check-target-mhpt4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:08:22.470742 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:22.470626 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq podName:b552dfdb-6037-4463-a58d-59a748531ee1 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:30.470608045 +0000 UTC m=+18.247625607 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hnwvq" (UniqueName: "kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq") pod "network-check-target-mhpt4" (UID: "b552dfdb-6037-4463-a58d-59a748531ee1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:08:22.772819 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:22.772366 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:22.772819 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:22.772487 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6z7f2" podUID="6a167d1c-5370-45bb-96c3-9deb3c9ddb50" Jul 02 19:08:22.772819 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:22.772587 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:22.772819 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:22.772706 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqszg" podUID="448316b6-cd2d-4c73-8ca6-103e4c64f18e" Jul 02 19:08:23.771587 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:23.771529 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:23.772140 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:23.771671 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhpt4" podUID="b552dfdb-6037-4463-a58d-59a748531ee1" Jul 02 19:08:24.773872 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:24.773566 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:24.773872 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:24.773722 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqszg" podUID="448316b6-cd2d-4c73-8ca6-103e4c64f18e" Jul 02 19:08:24.773872 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:24.773759 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:24.773872 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:24.773839 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6z7f2" podUID="6a167d1c-5370-45bb-96c3-9deb3c9ddb50" Jul 02 19:08:25.770811 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:25.770779 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:25.770997 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:25.770942 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhpt4" podUID="b552dfdb-6037-4463-a58d-59a748531ee1" Jul 02 19:08:26.770963 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:26.770925 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:26.770963 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:26.770972 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:26.771449 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:26.771072 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6z7f2" podUID="6a167d1c-5370-45bb-96c3-9deb3c9ddb50" Jul 02 19:08:26.771449 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:26.771200 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqszg" podUID="448316b6-cd2d-4c73-8ca6-103e4c64f18e" Jul 02 19:08:27.771341 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:27.771309 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:27.771773 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:27.771436 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhpt4" podUID="b552dfdb-6037-4463-a58d-59a748531ee1" Jul 02 19:08:28.771713 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:28.771682 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:28.772100 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:28.771807 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqszg" podUID="448316b6-cd2d-4c73-8ca6-103e4c64f18e" Jul 02 19:08:28.772100 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:28.771859 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:28.772100 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:28.771984 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6z7f2" podUID="6a167d1c-5370-45bb-96c3-9deb3c9ddb50" Jul 02 19:08:29.771489 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:29.771463 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:29.771742 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:29.771579 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhpt4" podUID="b552dfdb-6037-4463-a58d-59a748531ee1" Jul 02 19:08:30.334787 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:30.334704 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs\") pod \"network-metrics-daemon-6z7f2\" (UID: \"6a167d1c-5370-45bb-96c3-9deb3c9ddb50\") " pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:30.334910 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:30.334853 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:08:30.334944 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:30.334918 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs podName:6a167d1c-5370-45bb-96c3-9deb3c9ddb50 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:46.334902382 +0000 UTC m=+34.111919930 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs") pod "network-metrics-daemon-6z7f2" (UID: "6a167d1c-5370-45bb-96c3-9deb3c9ddb50") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:08:30.435304 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:30.435271 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret\") pod \"global-pull-secret-syncer-hqszg\" (UID: \"448316b6-cd2d-4c73-8ca6-103e4c64f18e\") " pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:30.435472 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:30.435449 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:08:30.435546 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:30.435532 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret podName:448316b6-cd2d-4c73-8ca6-103e4c64f18e nodeName:}" failed. No retries permitted until 2026-07-02 19:08:46.435511709 +0000 UTC m=+34.212529258 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret") pod "global-pull-secret-syncer-hqszg" (UID: "448316b6-cd2d-4c73-8ca6-103e4c64f18e") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:08:30.536141 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:30.536104 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnwvq\" (UniqueName: \"kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq\") pod \"network-check-target-mhpt4\" (UID: \"b552dfdb-6037-4463-a58d-59a748531ee1\") " pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:30.536294 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:30.536263 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:08:30.536294 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:30.536286 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:08:30.536379 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:30.536299 2562 projected.go:196] Error preparing data for projected volume kube-api-access-hnwvq for pod openshift-network-diagnostics/network-check-target-mhpt4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:08:30.536379 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:30.536352 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq podName:b552dfdb-6037-4463-a58d-59a748531ee1 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:46.536334561 +0000 UTC m=+34.313352108 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hnwvq" (UniqueName: "kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq") pod "network-check-target-mhpt4" (UID: "b552dfdb-6037-4463-a58d-59a748531ee1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:08:30.771288 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:30.771261 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:30.771439 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:30.771403 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6z7f2" podUID="6a167d1c-5370-45bb-96c3-9deb3c9ddb50" Jul 02 19:08:30.771481 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:30.771433 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:30.771565 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:30.771544 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqszg" podUID="448316b6-cd2d-4c73-8ca6-103e4c64f18e" Jul 02 19:08:31.771670 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:31.771626 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:31.772110 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:31.771768 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhpt4" podUID="b552dfdb-6037-4463-a58d-59a748531ee1" Jul 02 19:08:32.771761 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:32.771729 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:32.772146 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:32.771850 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqszg" podUID="448316b6-cd2d-4c73-8ca6-103e4c64f18e" Jul 02 19:08:32.772146 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:32.771919 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:32.772146 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:32.772015 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6z7f2" podUID="6a167d1c-5370-45bb-96c3-9deb3c9ddb50" Jul 02 19:08:33.770681 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:33.770652 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:33.770867 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:33.770756 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhpt4" podUID="b552dfdb-6037-4463-a58d-59a748531ee1" Jul 02 19:08:34.771444 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:34.771242 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:34.772092 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:34.771559 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6z7f2" podUID="6a167d1c-5370-45bb-96c3-9deb3c9ddb50" Jul 02 19:08:34.772092 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:34.771660 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:34.772092 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:34.771749 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqszg" podUID="448316b6-cd2d-4c73-8ca6-103e4c64f18e" Jul 02 19:08:34.883243 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:34.883209 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbph5" event={"ID":"7154ce25-e0a8-4d68-8a63-10b774b834cc","Type":"ContainerStarted","Data":"f5106f754f722b805ec56d22620a928ef3516448e77dd5a666961ddd3171d462"} Jul 02 19:08:34.885428 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:34.885399 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-75ftq" event={"ID":"270e3b43-4182-4af0-ba88-b327c157291e","Type":"ContainerStarted","Data":"5eec29acbb8aaf3d1bfac724e8017ec868aba50adabda96a920a8580a68dcb62"} Jul 02 19:08:34.887915 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:34.887542 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-45nlz" event={"ID":"c07acb28-0571-41c0-8a02-55cac407c73d","Type":"ContainerStarted","Data":"0897d56e6cb75a56d71159b90378d94f7c0160e4d9127dbd3afded9134e7d014"} Jul 02 19:08:34.890855 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:34.890834 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal" event={"ID":"634e6027337a88ddac8fcf1b6627c64d","Type":"ContainerStarted","Data":"8b0e27f6bb7cfa93c96ad3a22397f5e313204e0401707a3461ef39985d7957cb"} Jul 02 19:08:34.894015 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:34.893993 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" event={"ID":"f2b9ac49-b05c-4d63-95c2-14d0e77bb163","Type":"ContainerStarted","Data":"7cb6b553c7ed153f50ba6fd5e1d0324df2560dba294b5aa7f17943c0d6f1d613"} Jul 02 19:08:34.895429 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:34.895403 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xzqgm" event={"ID":"51243d8b-5625-444e-bc70-fa68e9b13ecb","Type":"ContainerStarted","Data":"3b7f599613b237078c8497f86769348c703d6bdfcd17aae7b9245dd5f201b54b"} Jul 02 19:08:34.897470 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:34.897214 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" event={"ID":"3c8d06c3-95ff-4490-9fdf-14c5570a0c52","Type":"ContainerStarted","Data":"6c34c871d4c001aaf84406369f0e3b8cf93baf89ade3d5e10b0975ed341f0048"} Jul 02 19:08:34.898981 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:34.898962 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d99qz" event={"ID":"28d07600-1581-4f70-a507-cd1cf2c8b2e2","Type":"ContainerStarted","Data":"e9e711e1ba20c91a4cae39cc9bb370aad787af28e9bd65059387a67f796af0ed"} Jul 02 19:08:34.900736 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:34.900709 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" event={"ID":"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891","Type":"ContainerStarted","Data":"941fda5fa713b43bf310b24009f75459b6296441d736b7216319ed4c90cd7c38"} Jul 02 19:08:34.913000 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:34.912901 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-147.ec2.internal" podStartSLOduration=20.912886155 podStartE2EDuration="20.912886155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:16.851546709 +0000 UTC m=+4.628564280" watchObservedRunningTime="2026-07-02 19:08:34.912886155 +0000 UTC m=+22.689903723" Jul 02 19:08:34.931737 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:34.931699 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xzqgm" podStartSLOduration=3.612697417 podStartE2EDuration="21.931685086s" totalImagesPullingTime="18.318987669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:34.931331561 +0000 UTC m=+22.708349131" watchObservedRunningTime="2026-07-02 19:08:34.931685086 +0000 UTC m=+22.708702652" Jul 02 19:08:34.950407 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:34.950364 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gnxkm" podStartSLOduration=3.586388282 podStartE2EDuration="21.950349889s" totalImagesPullingTime="18.363961607s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:34.949835768 +0000 UTC m=+22.726853347" watchObservedRunningTime="2026-07-02 19:08:34.950349889 +0000 UTC m=+22.727367473" Jul 02 19:08:34.970283 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:34.970242 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-d99qz" podStartSLOduration=2.283148951 podStartE2EDuration="21.970227193s" totalImagesPullingTime="19.687078242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:34.969909426 +0000 UTC m=+22.746927026" watchObservedRunningTime="2026-07-02 19:08:34.970227193 +0000 UTC m=+22.747244756" Jul 02 19:08:34.988747 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:34.988710 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-45nlz" podStartSLOduration=3.664320483 podStartE2EDuration="21.988696814s" totalImagesPullingTime="18.324376331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:34.9880731 +0000 UTC m=+22.765090755" watchObservedRunningTime="2026-07-02 19:08:34.988696814 +0000 UTC m=+22.765714381" Jul 02 19:08:35.005994 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:35.005957 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-147.ec2.internal" podStartSLOduration=21.005945316 podStartE2EDuration="21.005945316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:35.005778002 +0000 UTC m=+22.782795569" watchObservedRunningTime="2026-07-02 19:08:35.005945316 +0000 UTC m=+22.782962882" Jul 02 19:08:35.021046 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:35.021008 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-75ftq" podStartSLOduration=8.229504758 podStartE2EDuration="22.020994947s" totalImagesPullingTime="13.791490189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:35.020895673 +0000 UTC m=+22.797913240" watchObservedRunningTime="2026-07-02 19:08:35.020994947 +0000 UTC m=+22.798012513" Jul 02 19:08:35.101948 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:35.101923 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:35.770858 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:35.770825 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:35.771032 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:35.770944 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhpt4" podUID="b552dfdb-6037-4463-a58d-59a748531ee1" Jul 02 19:08:35.898788 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:35.898762 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:35.905968 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:35.905945 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" event={"ID":"f2b9ac49-b05c-4d63-95c2-14d0e77bb163","Type":"ContainerStarted","Data":"10492fc4c04f317dfac4cb72903e38b28d853efcbf96e9697f8ac6b604067def"} Jul 02 19:08:35.906084 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:35.906015 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" event={"ID":"f2b9ac49-b05c-4d63-95c2-14d0e77bb163","Type":"ContainerStarted","Data":"b0a0466e3881d9ba3c381722bdb76258ae32e32923a164fc52e67fedffff24ec"} Jul 02 19:08:35.906084 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:35.906053 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" event={"ID":"f2b9ac49-b05c-4d63-95c2-14d0e77bb163","Type":"ContainerStarted","Data":"ee93d7cd8d44fb4d2471dff8ab8a65f1fc0de9f9f6a38ec71e0f1da0470c0ed0"} Jul 02 19:08:35.906178 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:35.906094 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" event={"ID":"f2b9ac49-b05c-4d63-95c2-14d0e77bb163","Type":"ContainerStarted","Data":"e466011672f1f0d0ca9288a50a89ce385de2f7203ebc9503433100a39878c307"} Jul 02 19:08:35.906178 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:35.906124 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" event={"ID":"f2b9ac49-b05c-4d63-95c2-14d0e77bb163","Type":"ContainerStarted","Data":"f6fc0ae6905e4419cc344a64f653ebd9355cd298ebd438d8cec17efabfbc6844"} Jul 02 19:08:35.907235 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:35.907203 2562 generic.go:356] "Generic (PLEG): container finished" podID="7154ce25-e0a8-4d68-8a63-10b774b834cc" containerID="f5106f754f722b805ec56d22620a928ef3516448e77dd5a666961ddd3171d462" exitCode=0 Jul 02 19:08:35.907325 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:35.907231 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbph5" event={"ID":"7154ce25-e0a8-4d68-8a63-10b774b834cc","Type":"ContainerDied","Data":"f5106f754f722b805ec56d22620a928ef3516448e77dd5a666961ddd3171d462"} Jul 02 19:08:35.908792 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:35.908770 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5pm6h" event={"ID":"93122604-fc12-4759-a3ab-c12cd59219fa","Type":"ContainerStarted","Data":"4995f9add8417c135511ccc05229a6435bc2a8874f5f155afbf24c74e404e30c"} Jul 02 19:08:35.946036 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:35.945991 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5pm6h" podStartSLOduration=4.622922459 podStartE2EDuration="22.945978326s" totalImagesPullingTime="18.323055867s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:35.945905777 +0000 UTC m=+23.722923344" watchObservedRunningTime="2026-07-02 19:08:35.945978326 +0000 UTC m=+23.722995892" Jul 02 19:08:35.964269 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:35.964243 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:36.040533 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:36.040474 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:36.262411 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:36.262267 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 02 19:08:36.275013 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:36.274991 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-75ftq" Jul 02 19:08:36.275468 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:36.275453 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-75ftq" Jul 02 19:08:36.310114 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:36.310070 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:36.504541 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:36.504510 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:36.662973 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:36.662835 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T19:08:36.262407813Z","UUID":"c59ac989-f1ed-446a-ac86-8245b8e126f7","Handler":null,"Name":"","Endpoint":""} Jul 02 19:08:36.664227 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:36.664214 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 02 19:08:36.664303 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:36.664245 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 02 19:08:36.771219 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:36.771193 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:36.771416 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:36.771285 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqszg" podUID="448316b6-cd2d-4c73-8ca6-103e4c64f18e" Jul 02 19:08:36.771416 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:36.771390 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:36.771529 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:36.771508 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6z7f2" podUID="6a167d1c-5370-45bb-96c3-9deb3c9ddb50" Jul 02 19:08:36.913335 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:36.913249 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" event={"ID":"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891","Type":"ContainerStarted","Data":"cbae251afced31983cb657c81ea5de66185c8246aa4a51c28127a300018888f1"} Jul 02 19:08:36.913832 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:36.913435 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-75ftq" Jul 02 19:08:36.913947 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:36.913921 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-75ftq" Jul 02 19:08:36.959043 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:36.959015 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:37.073400 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:37.073364 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:37.509163 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:37.509131 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:37.770890 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:37.770867 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:37.771001 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:37.770983 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhpt4" podUID="b552dfdb-6037-4463-a58d-59a748531ee1" Jul 02 19:08:37.918835 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:37.918752 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" event={"ID":"f2b9ac49-b05c-4d63-95c2-14d0e77bb163","Type":"ContainerStarted","Data":"f4d1bf47f4be5de85a07cbdebf465d2dedb15d0b754a8d33ecf932447be3c028"} Jul 02 19:08:37.921315 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:37.921287 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" event={"ID":"7dd39a96-cf82-49eb-a56d-4b0e4cf5b891","Type":"ContainerStarted","Data":"b793e0a34c2ed36ebda8c077d982d400a878b3c39a7b5e90d49520b73597e159"} Jul 02 19:08:37.943683 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:37.943638 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-48svq" podStartSLOduration=6.997371715 podStartE2EDuration="24.943625975s" totalImagesPullingTime="17.94625426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:08:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:37.943269897 +0000 UTC m=+25.720287483" watchObservedRunningTime="2026-07-02 19:08:37.943625975 +0000 UTC m=+25.720643538" Jul 02 19:08:38.103953 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:38.103915 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:38.301328 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:38.301295 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:38.507180 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:38.507143 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:38.771498 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:38.771465 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:38.771696 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:38.771569 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6z7f2" podUID="6a167d1c-5370-45bb-96c3-9deb3c9ddb50" Jul 02 19:08:38.771696 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:38.771626 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:38.771825 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:38.771702 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqszg" podUID="448316b6-cd2d-4c73-8ca6-103e4c64f18e" Jul 02 19:08:38.908360 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:38.908329 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:39.578398 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:39.578278 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:39.770696 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:39.770672 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:39.770853 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:39.770770 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhpt4" podUID="b552dfdb-6037-4463-a58d-59a748531ee1" Jul 02 19:08:39.929383 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:39.929351 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" event={"ID":"f2b9ac49-b05c-4d63-95c2-14d0e77bb163","Type":"ContainerStarted","Data":"8b4d4639ee36a7db1c4c0387f109fbc3c74254dd7cdeddeaaa56f00bb96835b7"} Jul 02 19:08:39.929660 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:39.929634 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:39.945261 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:39.945237 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:39.957341 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:39.957302 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" podStartSLOduration=7.143583573 podStartE2EDuration="26.957289139s" totalImagesPullingTime="19.813705566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:39.956799937 +0000 UTC m=+27.733817527" watchObservedRunningTime="2026-07-02 19:08:39.957289139 +0000 UTC m=+27.734306708" Jul 02 19:08:39.990303 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:39.990268 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:40.098535 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:40.098467 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:40.212035 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:40.212007 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:40.307461 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:40.307425 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:40.510322 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:40.510284 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:40.714086 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:40.714001 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:40.771286 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:40.771257 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:40.771451 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:40.771302 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:40.771451 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:40.771387 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqszg" podUID="448316b6-cd2d-4c73-8ca6-103e4c64f18e" Jul 02 19:08:40.771581 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:40.771532 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6z7f2" podUID="6a167d1c-5370-45bb-96c3-9deb3c9ddb50" Jul 02 19:08:40.932520 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:40.932054 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:40.950615 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:40.950573 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:08:41.127930 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:41.127886 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:41.770831 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:41.770805 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:41.771270 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:41.770923 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhpt4" podUID="b552dfdb-6037-4463-a58d-59a748531ee1" Jul 02 19:08:42.772004 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:42.771973 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:42.772429 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:42.772092 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6z7f2" podUID="6a167d1c-5370-45bb-96c3-9deb3c9ddb50" Jul 02 19:08:42.772429 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:42.772400 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:42.772522 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:42.772486 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqszg" podUID="448316b6-cd2d-4c73-8ca6-103e4c64f18e" Jul 02 19:08:43.077029 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:43.076954 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:43.196001 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:43.195912 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:43.300137 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:43.300096 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:43.399206 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:43.399128 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:43.755099 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:43.755069 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:43.771245 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:43.771221 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:43.771357 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:43.771332 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhpt4" podUID="b552dfdb-6037-4463-a58d-59a748531ee1" Jul 02 19:08:43.855362 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:43.855334 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:43.940602 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:43.940566 2562 generic.go:356] "Generic (PLEG): container finished" podID="7154ce25-e0a8-4d68-8a63-10b774b834cc" containerID="97e3c7c80b2d0509c4c3fec075beb760e17ae5f3fea8b847dd65b7327b3d2503" exitCode=0 Jul 02 19:08:43.940700 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:43.940629 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbph5" event={"ID":"7154ce25-e0a8-4d68-8a63-10b774b834cc","Type":"ContainerDied","Data":"97e3c7c80b2d0509c4c3fec075beb760e17ae5f3fea8b847dd65b7327b3d2503"} Jul 02 19:08:44.160168 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:44.160017 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:44.217718 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:44.217696 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:44.370962 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:44.370941 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:44.470372 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:44.470314 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:44.760667 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:44.760584 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:44.771689 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:44.771666 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:44.771783 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:44.771768 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6z7f2" podUID="6a167d1c-5370-45bb-96c3-9deb3c9ddb50" Jul 02 19:08:44.771834 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:44.771820 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:44.771921 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:44.771905 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqszg" podUID="448316b6-cd2d-4c73-8ca6-103e4c64f18e" Jul 02 19:08:44.974583 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:44.974556 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:45.189692 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:45.189664 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:45.771252 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:45.771233 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:45.771387 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:45.771323 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhpt4" podUID="b552dfdb-6037-4463-a58d-59a748531ee1" Jul 02 19:08:45.946072 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:45.946046 2562 generic.go:356] "Generic (PLEG): container finished" podID="7154ce25-e0a8-4d68-8a63-10b774b834cc" containerID="435e82589a0ea2fade2b1c2799bfc433e58b8e7199dcfce494b3cdb5d589f8a4" exitCode=0 Jul 02 19:08:45.946213 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:45.946110 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbph5" event={"ID":"7154ce25-e0a8-4d68-8a63-10b774b834cc","Type":"ContainerDied","Data":"435e82589a0ea2fade2b1c2799bfc433e58b8e7199dcfce494b3cdb5d589f8a4"} Jul 02 19:08:46.181752 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:46.181675 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:46.233698 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:46.233668 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:46.356009 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:46.355978 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs\") pod \"network-metrics-daemon-6z7f2\" (UID: \"6a167d1c-5370-45bb-96c3-9deb3c9ddb50\") " pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:46.356159 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:46.356097 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:08:46.356159 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:46.356158 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs podName:6a167d1c-5370-45bb-96c3-9deb3c9ddb50 nodeName:}" failed. No retries permitted until 2026-07-02 19:09:18.356141511 +0000 UTC m=+66.133159056 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs") pod "network-metrics-daemon-6z7f2" (UID: "6a167d1c-5370-45bb-96c3-9deb3c9ddb50") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:08:46.457019 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:46.456957 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret\") pod \"global-pull-secret-syncer-hqszg\" (UID: \"448316b6-cd2d-4c73-8ca6-103e4c64f18e\") " pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:46.457128 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:46.457058 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:08:46.457128 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:46.457100 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret podName:448316b6-cd2d-4c73-8ca6-103e4c64f18e nodeName:}" failed. No retries permitted until 2026-07-02 19:09:18.457088665 +0000 UTC m=+66.234106210 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret") pod "global-pull-secret-syncer-hqszg" (UID: "448316b6-cd2d-4c73-8ca6-103e4c64f18e") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:08:46.558135 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:46.558102 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnwvq\" (UniqueName: \"kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq\") pod \"network-check-target-mhpt4\" (UID: \"b552dfdb-6037-4463-a58d-59a748531ee1\") " pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:46.558245 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:46.558232 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:08:46.558292 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:46.558251 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:08:46.558292 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:46.558260 2562 projected.go:196] Error preparing data for projected volume kube-api-access-hnwvq for pod openshift-network-diagnostics/network-check-target-mhpt4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:08:46.558363 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:46.558302 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq podName:b552dfdb-6037-4463-a58d-59a748531ee1 nodeName:}" failed. No retries permitted until 2026-07-02 19:09:18.558290112 +0000 UTC m=+66.335307656 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hnwvq" (UniqueName: "kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq") pod "network-check-target-mhpt4" (UID: "b552dfdb-6037-4463-a58d-59a748531ee1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:08:46.770752 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:46.770725 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:46.770873 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:46.770844 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6z7f2" podUID="6a167d1c-5370-45bb-96c3-9deb3c9ddb50" Jul 02 19:08:46.770915 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:46.770901 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:46.771008 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:46.770990 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqszg" podUID="448316b6-cd2d-4c73-8ca6-103e4c64f18e" Jul 02 19:08:47.771101 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:47.771075 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:47.771510 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:47.771187 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhpt4" podUID="b552dfdb-6037-4463-a58d-59a748531ee1" Jul 02 19:08:47.952400 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:47.952371 2562 generic.go:356] "Generic (PLEG): container finished" podID="7154ce25-e0a8-4d68-8a63-10b774b834cc" containerID="902da48906a90510ccea313d091fc47df5dedc5bdb0ec5c5894b2b2dc89ca345" exitCode=0 Jul 02 19:08:47.952529 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:47.952410 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbph5" event={"ID":"7154ce25-e0a8-4d68-8a63-10b774b834cc","Type":"ContainerDied","Data":"902da48906a90510ccea313d091fc47df5dedc5bdb0ec5c5894b2b2dc89ca345"} Jul 02 19:08:48.167048 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:48.166976 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:48.308254 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:48.308232 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:48.400899 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:48.400872 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:48.582648 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:48.582620 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:48.642951 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:48.642922 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:48.770869 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:48.770843 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:48.771014 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:48.770943 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqszg" podUID="448316b6-cd2d-4c73-8ca6-103e4c64f18e" Jul 02 19:08:48.771073 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:48.771029 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:48.771171 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:48.771133 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6z7f2" podUID="6a167d1c-5370-45bb-96c3-9deb3c9ddb50" Jul 02 19:08:49.112988 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:49.112951 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:49.540587 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:49.540556 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:49.631670 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:49.631640 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:49.770722 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:49.770693 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:49.770887 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:49.770821 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhpt4" podUID="b552dfdb-6037-4463-a58d-59a748531ee1" Jul 02 19:08:49.895563 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:49.895468 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mhpt4"] Jul 02 19:08:49.896353 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:49.896330 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hqszg"] Jul 02 19:08:49.896445 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:49.896432 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:49.896755 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:49.896545 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqszg" podUID="448316b6-cd2d-4c73-8ca6-103e4c64f18e" Jul 02 19:08:49.910357 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:49.910330 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6z7f2"] Jul 02 19:08:49.910495 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:49.910439 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:49.910575 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:49.910553 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6z7f2" podUID="6a167d1c-5370-45bb-96c3-9deb3c9ddb50" Jul 02 19:08:49.958033 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:49.958001 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:49.958193 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:49.958104 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhpt4" podUID="b552dfdb-6037-4463-a58d-59a748531ee1" Jul 02 19:08:51.771153 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:51.771116 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:51.771557 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:51.771127 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:51.771557 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:51.771238 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:51.771557 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:51.771240 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhpt4" podUID="b552dfdb-6037-4463-a58d-59a748531ee1" Jul 02 19:08:51.771557 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:51.771314 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6z7f2" podUID="6a167d1c-5370-45bb-96c3-9deb3c9ddb50" Jul 02 19:08:51.771557 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:51.771397 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqszg" podUID="448316b6-cd2d-4c73-8ca6-103e4c64f18e" Jul 02 19:08:53.771546 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:53.771366 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:53.772044 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:53.771366 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:53.772044 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:53.771655 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hqszg" podUID="448316b6-cd2d-4c73-8ca6-103e4c64f18e" Jul 02 19:08:53.772044 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:53.771386 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:53.772044 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:53.771753 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6z7f2" podUID="6a167d1c-5370-45bb-96c3-9deb3c9ddb50" Jul 02 19:08:53.772044 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:53.771787 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhpt4" podUID="b552dfdb-6037-4463-a58d-59a748531ee1" Jul 02 19:08:53.969749 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:53.969664 2562 generic.go:356] "Generic (PLEG): container finished" podID="7154ce25-e0a8-4d68-8a63-10b774b834cc" containerID="9487f69bfd5295bf9bdb8e804337fd7fc1822f871d3033375586b6cfbfc11fdd" exitCode=0 Jul 02 19:08:53.969749 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:53.969717 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbph5" event={"ID":"7154ce25-e0a8-4d68-8a63-10b774b834cc","Type":"ContainerDied","Data":"9487f69bfd5295bf9bdb8e804337fd7fc1822f871d3033375586b6cfbfc11fdd"} Jul 02 19:08:54.505206 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.505185 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-147.ec2.internal" event="NodeReady" Jul 02 19:08:54.505335 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.505280 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 02 19:08:54.562961 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.562496 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2mrvq"] Jul 02 19:08:54.582754 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.582734 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-v7lc8"] Jul 02 19:08:54.582876 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.582849 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2mrvq" Jul 02 19:08:54.585072 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.585053 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 02 19:08:54.585303 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.585261 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 02 19:08:54.585459 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.585437 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 02 19:08:54.585531 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.585481 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pjv9d\"" Jul 02 19:08:54.606469 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.606444 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2mrvq"] Jul 02 19:08:54.606559 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.606481 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v7lc8"] Jul 02 19:08:54.606559 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.606516 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v7lc8" Jul 02 19:08:54.606908 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.606884 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:54.609141 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.609128 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 02 19:08:54.609224 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.609165 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 02 19:08:54.609268 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.609253 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 02 19:08:54.609328 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.609257 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jjnz9\"" Jul 02 19:08:54.609407 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.609388 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 02 19:08:54.656269 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.656248 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9pdxg"] Jul 02 19:08:54.676278 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.676260 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9pdxg"] Jul 02 19:08:54.676370 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.676358 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9pdxg" Jul 02 19:08:54.678970 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.678944 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 02 19:08:54.679075 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.678993 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-45sjn\"" Jul 02 19:08:54.679075 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.678993 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 02 19:08:54.697083 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.697057 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:54.717502 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.717481 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwvwl\" (UniqueName: \"kubernetes.io/projected/3e079bf3-979e-4a34-9f19-8c6633bd0a15-kube-api-access-cwvwl\") pod \"ingress-canary-2mrvq\" (UID: \"3e079bf3-979e-4a34-9f19-8c6633bd0a15\") " pod="openshift-ingress-canary/ingress-canary-2mrvq" Jul 02 19:08:54.717566 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.717510 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0499397a-c856-416b-b3e2-3b3998080ea7-data-volume\") pod \"insights-runtime-extractor-v7lc8\" (UID: \"0499397a-c856-416b-b3e2-3b3998080ea7\") " pod="openshift-insights/insights-runtime-extractor-v7lc8" Jul 02 19:08:54.717566 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.717526 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0499397a-c856-416b-b3e2-3b3998080ea7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v7lc8\" (UID: \"0499397a-c856-416b-b3e2-3b3998080ea7\") " pod="openshift-insights/insights-runtime-extractor-v7lc8" Jul 02 19:08:54.717657 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.717613 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0499397a-c856-416b-b3e2-3b3998080ea7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v7lc8\" (UID: \"0499397a-c856-416b-b3e2-3b3998080ea7\") " pod="openshift-insights/insights-runtime-extractor-v7lc8" Jul 02 19:08:54.717690 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.717656 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e079bf3-979e-4a34-9f19-8c6633bd0a15-cert\") pod \"ingress-canary-2mrvq\" (UID: \"3e079bf3-979e-4a34-9f19-8c6633bd0a15\") " pod="openshift-ingress-canary/ingress-canary-2mrvq" Jul 02 19:08:54.717690 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.717674 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqz4n\" (UniqueName: \"kubernetes.io/projected/0499397a-c856-416b-b3e2-3b3998080ea7-kube-api-access-nqz4n\") pod \"insights-runtime-extractor-v7lc8\" (UID: \"0499397a-c856-416b-b3e2-3b3998080ea7\") " pod="openshift-insights/insights-runtime-extractor-v7lc8" Jul 02 19:08:54.717788 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.717734 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0499397a-c856-416b-b3e2-3b3998080ea7-crio-socket\") pod \"insights-runtime-extractor-v7lc8\" (UID: \"0499397a-c856-416b-b3e2-3b3998080ea7\") " pod="openshift-insights/insights-runtime-extractor-v7lc8" Jul 02 19:08:54.772950 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.772926 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:54.778782 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.778761 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vrmvk"] Jul 02 19:08:54.800371 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.800345 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vrmvk"] Jul 02 19:08:54.800455 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.800397 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vrmvk" Jul 02 19:08:54.803025 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.803009 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 02 19:08:54.803146 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.803130 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-blsrw\"" Jul 02 19:08:54.818715 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.818692 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b070f3c1-e648-48ab-9af8-348a82ea5404-config-volume\") pod \"dns-default-9pdxg\" (UID: \"b070f3c1-e648-48ab-9af8-348a82ea5404\") " pod="openshift-dns/dns-default-9pdxg" Jul 02 19:08:54.818847 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.818734 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0499397a-c856-416b-b3e2-3b3998080ea7-crio-socket\") pod \"insights-runtime-extractor-v7lc8\" (UID: \"0499397a-c856-416b-b3e2-3b3998080ea7\") " pod="openshift-insights/insights-runtime-extractor-v7lc8" Jul 02 19:08:54.818847 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.818755 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b070f3c1-e648-48ab-9af8-348a82ea5404-metrics-tls\") pod \"dns-default-9pdxg\" (UID: \"b070f3c1-e648-48ab-9af8-348a82ea5404\") " pod="openshift-dns/dns-default-9pdxg" Jul 02 19:08:54.818847 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.818773 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwvwl\" (UniqueName: \"kubernetes.io/projected/3e079bf3-979e-4a34-9f19-8c6633bd0a15-kube-api-access-cwvwl\") pod \"ingress-canary-2mrvq\" (UID: \"3e079bf3-979e-4a34-9f19-8c6633bd0a15\") " pod="openshift-ingress-canary/ingress-canary-2mrvq" Jul 02 19:08:54.818847 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.818825 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0499397a-c856-416b-b3e2-3b3998080ea7-data-volume\") pod \"insights-runtime-extractor-v7lc8\" (UID: \"0499397a-c856-416b-b3e2-3b3998080ea7\") " pod="openshift-insights/insights-runtime-extractor-v7lc8" Jul 02 19:08:54.819044 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.818857 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0499397a-c856-416b-b3e2-3b3998080ea7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v7lc8\" (UID: \"0499397a-c856-416b-b3e2-3b3998080ea7\") " pod="openshift-insights/insights-runtime-extractor-v7lc8" Jul 02 19:08:54.819044 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.818944 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0499397a-c856-416b-b3e2-3b3998080ea7-crio-socket\") pod \"insights-runtime-extractor-v7lc8\" (UID: \"0499397a-c856-416b-b3e2-3b3998080ea7\") " pod="openshift-insights/insights-runtime-extractor-v7lc8" Jul 02 19:08:54.819044 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.818947 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0499397a-c856-416b-b3e2-3b3998080ea7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v7lc8\" (UID: \"0499397a-c856-416b-b3e2-3b3998080ea7\") " pod="openshift-insights/insights-runtime-extractor-v7lc8" Jul 02 19:08:54.819044 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.819013 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e079bf3-979e-4a34-9f19-8c6633bd0a15-cert\") pod \"ingress-canary-2mrvq\" (UID: \"3e079bf3-979e-4a34-9f19-8c6633bd0a15\") " pod="openshift-ingress-canary/ingress-canary-2mrvq" Jul 02 19:08:54.819044 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.819040 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqz4n\" (UniqueName: \"kubernetes.io/projected/0499397a-c856-416b-b3e2-3b3998080ea7-kube-api-access-nqz4n\") pod \"insights-runtime-extractor-v7lc8\" (UID: \"0499397a-c856-416b-b3e2-3b3998080ea7\") " pod="openshift-insights/insights-runtime-extractor-v7lc8" Jul 02 19:08:54.819267 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.819072 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b070f3c1-e648-48ab-9af8-348a82ea5404-tmp-dir\") pod \"dns-default-9pdxg\" (UID: \"b070f3c1-e648-48ab-9af8-348a82ea5404\") " pod="openshift-dns/dns-default-9pdxg" Jul 02 19:08:54.819267 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.819097 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv5x6\" (UniqueName: \"kubernetes.io/projected/b070f3c1-e648-48ab-9af8-348a82ea5404-kube-api-access-kv5x6\") pod \"dns-default-9pdxg\" (UID: \"b070f3c1-e648-48ab-9af8-348a82ea5404\") " pod="openshift-dns/dns-default-9pdxg" Jul 02 19:08:54.819267 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.819185 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0499397a-c856-416b-b3e2-3b3998080ea7-data-volume\") pod \"insights-runtime-extractor-v7lc8\" (UID: \"0499397a-c856-416b-b3e2-3b3998080ea7\") " pod="openshift-insights/insights-runtime-extractor-v7lc8" Jul 02 19:08:54.819417 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.819399 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0499397a-c856-416b-b3e2-3b3998080ea7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-v7lc8\" (UID: \"0499397a-c856-416b-b3e2-3b3998080ea7\") " pod="openshift-insights/insights-runtime-extractor-v7lc8" Jul 02 19:08:54.822948 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.822919 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0499397a-c856-416b-b3e2-3b3998080ea7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-v7lc8\" (UID: \"0499397a-c856-416b-b3e2-3b3998080ea7\") " pod="openshift-insights/insights-runtime-extractor-v7lc8" Jul 02 19:08:54.823031 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.822988 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e079bf3-979e-4a34-9f19-8c6633bd0a15-cert\") pod \"ingress-canary-2mrvq\" (UID: \"3e079bf3-979e-4a34-9f19-8c6633bd0a15\") " pod="openshift-ingress-canary/ingress-canary-2mrvq" Jul 02 19:08:54.826602 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.826551 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwvwl\" (UniqueName: \"kubernetes.io/projected/3e079bf3-979e-4a34-9f19-8c6633bd0a15-kube-api-access-cwvwl\") pod \"ingress-canary-2mrvq\" (UID: \"3e079bf3-979e-4a34-9f19-8c6633bd0a15\") " pod="openshift-ingress-canary/ingress-canary-2mrvq" Jul 02 19:08:54.827452 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.827431 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqz4n\" (UniqueName: \"kubernetes.io/projected/0499397a-c856-416b-b3e2-3b3998080ea7-kube-api-access-nqz4n\") pod \"insights-runtime-extractor-v7lc8\" (UID: \"0499397a-c856-416b-b3e2-3b3998080ea7\") " pod="openshift-insights/insights-runtime-extractor-v7lc8" Jul 02 19:08:54.832518 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.832493 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:54.892214 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.892191 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2mrvq" Jul 02 19:08:54.914781 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.914764 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-v7lc8" Jul 02 19:08:54.920008 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.919989 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9c68493d-8279-4226-9970-d63f8dba9acb-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-vrmvk\" (UID: \"9c68493d-8279-4226-9970-d63f8dba9acb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vrmvk" Jul 02 19:08:54.920100 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.920018 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b070f3c1-e648-48ab-9af8-348a82ea5404-config-volume\") pod \"dns-default-9pdxg\" (UID: \"b070f3c1-e648-48ab-9af8-348a82ea5404\") " pod="openshift-dns/dns-default-9pdxg" Jul 02 19:08:54.920100 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.920047 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b070f3c1-e648-48ab-9af8-348a82ea5404-metrics-tls\") pod \"dns-default-9pdxg\" (UID: \"b070f3c1-e648-48ab-9af8-348a82ea5404\") " pod="openshift-dns/dns-default-9pdxg" Jul 02 19:08:54.920211 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.920118 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b070f3c1-e648-48ab-9af8-348a82ea5404-tmp-dir\") pod \"dns-default-9pdxg\" (UID: \"b070f3c1-e648-48ab-9af8-348a82ea5404\") " pod="openshift-dns/dns-default-9pdxg" Jul 02 19:08:54.920211 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.920134 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv5x6\" (UniqueName: \"kubernetes.io/projected/b070f3c1-e648-48ab-9af8-348a82ea5404-kube-api-access-kv5x6\") pod \"dns-default-9pdxg\" (UID: \"b070f3c1-e648-48ab-9af8-348a82ea5404\") " pod="openshift-dns/dns-default-9pdxg" Jul 02 19:08:54.920451 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.920429 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b070f3c1-e648-48ab-9af8-348a82ea5404-tmp-dir\") pod \"dns-default-9pdxg\" (UID: \"b070f3c1-e648-48ab-9af8-348a82ea5404\") " pod="openshift-dns/dns-default-9pdxg" Jul 02 19:08:54.920553 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.920536 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b070f3c1-e648-48ab-9af8-348a82ea5404-config-volume\") pod \"dns-default-9pdxg\" (UID: \"b070f3c1-e648-48ab-9af8-348a82ea5404\") " pod="openshift-dns/dns-default-9pdxg" Jul 02 19:08:54.922160 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.922138 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b070f3c1-e648-48ab-9af8-348a82ea5404-metrics-tls\") pod \"dns-default-9pdxg\" (UID: \"b070f3c1-e648-48ab-9af8-348a82ea5404\") " pod="openshift-dns/dns-default-9pdxg" Jul 02 19:08:54.928732 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.928715 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv5x6\" (UniqueName: \"kubernetes.io/projected/b070f3c1-e648-48ab-9af8-348a82ea5404-kube-api-access-kv5x6\") pod \"dns-default-9pdxg\" (UID: \"b070f3c1-e648-48ab-9af8-348a82ea5404\") " pod="openshift-dns/dns-default-9pdxg" Jul 02 19:08:54.976281 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.976253 2562 generic.go:356] "Generic (PLEG): container finished" podID="7154ce25-e0a8-4d68-8a63-10b774b834cc" containerID="ef10e1cfe8118e5976143fe03e852bb0b9271a7673a6a8a190405d4f37ad28a7" exitCode=0 Jul 02 19:08:54.976397 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.976311 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbph5" event={"ID":"7154ce25-e0a8-4d68-8a63-10b774b834cc","Type":"ContainerDied","Data":"ef10e1cfe8118e5976143fe03e852bb0b9271a7673a6a8a190405d4f37ad28a7"} Jul 02 19:08:54.984140 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:54.984121 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9pdxg" Jul 02 19:08:55.024885 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.022429 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9c68493d-8279-4226-9970-d63f8dba9acb-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-vrmvk\" (UID: \"9c68493d-8279-4226-9970-d63f8dba9acb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vrmvk" Jul 02 19:08:55.024885 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:55.022553 2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 02 19:08:55.024885 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:08:55.022625 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c68493d-8279-4226-9970-d63f8dba9acb-tls-certificates podName:9c68493d-8279-4226-9970-d63f8dba9acb nodeName:}" failed. No retries permitted until 2026-07-02 19:08:55.522606341 +0000 UTC m=+43.299623917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/9c68493d-8279-4226-9970-d63f8dba9acb-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-vrmvk" (UID: "9c68493d-8279-4226-9970-d63f8dba9acb") : secret "prometheus-operator-admission-webhook-tls" not found Jul 02 19:08:55.115989 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.115795 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2mrvq"] Jul 02 19:08:55.119207 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.119180 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-v7lc8"] Jul 02 19:08:55.119683 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:55.119583 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e079bf3_979e_4a34_9f19_8c6633bd0a15.slice/crio-8beac83a6e45fbbee27b0cdd5ea758c5981b33061a55db840093133d4eb19014 WatchSource:0}: Error finding container 8beac83a6e45fbbee27b0cdd5ea758c5981b33061a55db840093133d4eb19014: Status 404 returned error can't find the container with id 8beac83a6e45fbbee27b0cdd5ea758c5981b33061a55db840093133d4eb19014 Jul 02 19:08:55.123944 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:55.123906 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0499397a_c856_416b_b3e2_3b3998080ea7.slice/crio-bc483dd854d29e7cf38f7efbffb072bd4b8e579700827aa88aaf7214818a2aff WatchSource:0}: Error finding container bc483dd854d29e7cf38f7efbffb072bd4b8e579700827aa88aaf7214818a2aff: Status 404 returned error can't find the container with id bc483dd854d29e7cf38f7efbffb072bd4b8e579700827aa88aaf7214818a2aff Jul 02 19:08:55.156813 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.156788 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:55.169743 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.169719 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9pdxg"] Jul 02 19:08:55.174173 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:55.174137 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb070f3c1_e648_48ab_9af8_348a82ea5404.slice/crio-83109e892f33b560248b676f13abdb797c9220fc9df509ee175bb723b74f31f5 WatchSource:0}: Error finding container 83109e892f33b560248b676f13abdb797c9220fc9df509ee175bb723b74f31f5: Status 404 returned error can't find the container with id 83109e892f33b560248b676f13abdb797c9220fc9df509ee175bb723b74f31f5 Jul 02 19:08:55.374269 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.374204 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:55.448110 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.448084 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:55.526279 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.526252 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9c68493d-8279-4226-9970-d63f8dba9acb-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-vrmvk\" (UID: \"9c68493d-8279-4226-9970-d63f8dba9acb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vrmvk" Jul 02 19:08:55.529969 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.529946 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9c68493d-8279-4226-9970-d63f8dba9acb-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-vrmvk\" (UID: \"9c68493d-8279-4226-9970-d63f8dba9acb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vrmvk" Jul 02 19:08:55.710987 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.710622 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vrmvk" Jul 02 19:08:55.771548 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.771095 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:08:55.771548 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.771271 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:08:55.771548 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.771289 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:08:55.774744 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.774430 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 19:08:55.775726 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.775540 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r7xgd\"" Jul 02 19:08:55.775726 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.775569 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 19:08:55.775881 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.775768 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 19:08:55.775881 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.775853 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5rf5h\"" Jul 02 19:08:55.775988 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.775942 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 19:08:55.804275 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.797865 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-846c85d97c-qpn8m"] Jul 02 19:08:55.820835 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.820747 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:55.828639 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.828329 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-846c85d97c-qpn8m"] Jul 02 19:08:55.828639 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.828432 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:55.831857 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.831828 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 02 19:08:55.832147 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.832126 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kq4f6\"" Jul 02 19:08:55.832348 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.832325 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 02 19:08:55.832570 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.832553 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 02 19:08:55.832814 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.832799 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 02 19:08:55.832994 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.832980 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 02 19:08:55.834863 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.834846 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 02 19:08:55.835069 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.835054 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 02 19:08:55.840798 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.840777 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 02 19:08:55.883688 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.883556 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:55.917335 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.917306 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vrmvk"] Jul 02 19:08:55.929185 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.929163 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4fa7416-7a4f-4371-a663-46861e8e84e8-console-oauth-config\") pod \"console-846c85d97c-qpn8m\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:55.929334 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.929192 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-oauth-serving-cert\") pod \"console-846c85d97c-qpn8m\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:55.929334 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.929212 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-service-ca\") pod \"console-846c85d97c-qpn8m\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:55.929334 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.929239 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-trusted-ca-bundle\") pod \"console-846c85d97c-qpn8m\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:55.929334 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.929311 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzvbk\" (UniqueName: \"kubernetes.io/projected/d4fa7416-7a4f-4371-a663-46861e8e84e8-kube-api-access-bzvbk\") pod \"console-846c85d97c-qpn8m\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:55.929536 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.929484 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4fa7416-7a4f-4371-a663-46861e8e84e8-console-serving-cert\") pod \"console-846c85d97c-qpn8m\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:55.929536 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.929521 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-console-config\") pod \"console-846c85d97c-qpn8m\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:55.989128 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.988995 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vrmvk" event={"ID":"9c68493d-8279-4226-9970-d63f8dba9acb","Type":"ContainerStarted","Data":"5a988ae3ac3c08a7fcab37aed303253bd58463f70b0f5480b3f9f3bf3b156fe5"} Jul 02 19:08:55.990884 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.990854 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9pdxg" event={"ID":"b070f3c1-e648-48ab-9af8-348a82ea5404","Type":"ContainerStarted","Data":"83109e892f33b560248b676f13abdb797c9220fc9df509ee175bb723b74f31f5"} Jul 02 19:08:55.992305 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.992274 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v7lc8" event={"ID":"0499397a-c856-416b-b3e2-3b3998080ea7","Type":"ContainerStarted","Data":"bc483dd854d29e7cf38f7efbffb072bd4b8e579700827aa88aaf7214818a2aff"} Jul 02 19:08:55.994028 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:55.993923 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2mrvq" event={"ID":"3e079bf3-979e-4a34-9f19-8c6633bd0a15","Type":"ContainerStarted","Data":"8beac83a6e45fbbee27b0cdd5ea758c5981b33061a55db840093133d4eb19014"} Jul 02 19:08:56.002513 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:56.001794 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbph5" event={"ID":"7154ce25-e0a8-4d68-8a63-10b774b834cc","Type":"ContainerStarted","Data":"790d33d41a9f79c3cc87b0fdeeb34286ea7758e7376b74bc37ed887f7231dbeb"} Jul 02 19:08:56.023866 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:56.023818 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tbph5" podStartSLOduration=8.391974604 podStartE2EDuration="43.02380481s" totalImagesPullingTime="34.631830206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:08:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:56.023086208 +0000 UTC m=+43.800103814" watchObservedRunningTime="2026-07-02 19:08:56.02380481 +0000 UTC m=+43.800822380" Jul 02 19:08:56.032227 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:56.029823 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-console-config\") pod \"console-846c85d97c-qpn8m\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:56.032227 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:56.029869 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4fa7416-7a4f-4371-a663-46861e8e84e8-console-oauth-config\") pod \"console-846c85d97c-qpn8m\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:56.032227 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:56.029896 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-oauth-serving-cert\") pod \"console-846c85d97c-qpn8m\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:56.032227 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:56.029926 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-service-ca\") pod \"console-846c85d97c-qpn8m\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:56.032227 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:56.029958 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-trusted-ca-bundle\") pod \"console-846c85d97c-qpn8m\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:56.032227 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:56.029996 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzvbk\" (UniqueName: \"kubernetes.io/projected/d4fa7416-7a4f-4371-a663-46861e8e84e8-kube-api-access-bzvbk\") pod \"console-846c85d97c-qpn8m\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:56.032227 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:56.030087 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4fa7416-7a4f-4371-a663-46861e8e84e8-console-serving-cert\") pod \"console-846c85d97c-qpn8m\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:56.032227 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:56.031321 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-oauth-serving-cert\") pod \"console-846c85d97c-qpn8m\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:56.032227 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:56.031926 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-console-config\") pod \"console-846c85d97c-qpn8m\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:56.032799 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:56.032766 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-service-ca\") pod \"console-846c85d97c-qpn8m\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:56.033576 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:56.033551 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-trusted-ca-bundle\") pod \"console-846c85d97c-qpn8m\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:56.039168 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:56.039142 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4fa7416-7a4f-4371-a663-46861e8e84e8-console-serving-cert\") pod \"console-846c85d97c-qpn8m\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:56.039265 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:56.039241 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4fa7416-7a4f-4371-a663-46861e8e84e8-console-oauth-config\") pod \"console-846c85d97c-qpn8m\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:56.042307 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:56.042279 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzvbk\" (UniqueName: \"kubernetes.io/projected/d4fa7416-7a4f-4371-a663-46861e8e84e8-kube-api-access-bzvbk\") pod \"console-846c85d97c-qpn8m\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:56.144205 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:56.144126 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:56.166114 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:56.166089 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:08:56.303544 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:56.303484 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:56.351865 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:56.351773 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-846c85d97c-qpn8m"] Jul 02 19:08:56.357626 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:08:56.357519 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fa7416_7a4f_4371_a663_46861e8e84e8.slice/crio-2b29b979e85496b8d648f5c42085ad2265d17487e938cbe30c8f34aed63687bd WatchSource:0}: Error finding container 2b29b979e85496b8d648f5c42085ad2265d17487e938cbe30c8f34aed63687bd: Status 404 returned error can't find the container with id 2b29b979e85496b8d648f5c42085ad2265d17487e938cbe30c8f34aed63687bd Jul 02 19:08:57.008464 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:57.008430 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846c85d97c-qpn8m" event={"ID":"d4fa7416-7a4f-4371-a663-46861e8e84e8","Type":"ContainerStarted","Data":"2b29b979e85496b8d648f5c42085ad2265d17487e938cbe30c8f34aed63687bd"} Jul 02 19:08:59.188842 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:59.188797 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:59.290903 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:08:59.290878 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:03.386189 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:03.386004 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:03.507370 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:03.507342 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:04.030879 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.030840 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vrmvk" event={"ID":"9c68493d-8279-4226-9970-d63f8dba9acb","Type":"ContainerStarted","Data":"e2a902ea3f3f8720d200db398541f22e4579f3ae0dbad0402864054f76230cec"} Jul 02 19:09:04.031084 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.031038 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vrmvk" Jul 02 19:09:04.032902 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.032872 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9pdxg" event={"ID":"b070f3c1-e648-48ab-9af8-348a82ea5404","Type":"ContainerStarted","Data":"bfef0523f591d952be01407f87fe04bbbf16ddb614d90415783ab18b03b7da77"} Jul 02 19:09:04.033064 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.032930 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9pdxg" Jul 02 19:09:04.033064 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.032954 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9pdxg" event={"ID":"b070f3c1-e648-48ab-9af8-348a82ea5404","Type":"ContainerStarted","Data":"f108463a5576544f25be7091d65059fce775cecdd2439abb25a152757cb7d1d6"} Jul 02 19:09:04.034621 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.034561 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v7lc8" event={"ID":"0499397a-c856-416b-b3e2-3b3998080ea7","Type":"ContainerStarted","Data":"a6bdcf69ce87fafc6a10c149b38f46df9011af45eb50644eba8b73b18f0c0297"} Jul 02 19:09:04.036373 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.036346 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2mrvq" event={"ID":"3e079bf3-979e-4a34-9f19-8c6633bd0a15","Type":"ContainerStarted","Data":"2ec38bf31795fabc5e2d4df32f913524be860ad940ce56b24a46005f80e0b968"} Jul 02 19:09:04.036947 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.036927 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vrmvk" Jul 02 19:09:04.038612 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.038565 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846c85d97c-qpn8m" event={"ID":"d4fa7416-7a4f-4371-a663-46861e8e84e8","Type":"ContainerStarted","Data":"873fa6fb0ec71e8670941d7b87ef0714158244b2393a6f78d7fe951b9f6d96ee"} Jul 02 19:09:04.046904 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.046862 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vrmvk" podStartSLOduration=2.6329120169999998 podStartE2EDuration="10.046845844s" totalImagesPullingTime="7.413933827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:09:04.045545354 +0000 UTC m=+51.822562939" watchObservedRunningTime="2026-07-02 19:09:04.046845844 +0000 UTC m=+51.823863447" Jul 02 19:09:04.059870 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.059835 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2mrvq" podStartSLOduration=1.844273837 podStartE2EDuration="10.059821817s" totalImagesPullingTime="8.21554798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:09:04.05926495 +0000 UTC m=+51.836282536" watchObservedRunningTime="2026-07-02 19:09:04.059821817 +0000 UTC m=+51.836839383" Jul 02 19:09:04.075718 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.075680 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-846c85d97c-qpn8m" podStartSLOduration=2.069100555 podStartE2EDuration="9.075666318s" totalImagesPullingTime="7.006565763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:09:04.074728861 +0000 UTC m=+51.851746429" watchObservedRunningTime="2026-07-02 19:09:04.075666318 +0000 UTC m=+51.852683885" Jul 02 19:09:04.093462 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.093409 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9pdxg" podStartSLOduration=2.279462544 podStartE2EDuration="10.093389818s" totalImagesPullingTime="7.813927274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:09:04.092003545 +0000 UTC m=+51.869021110" watchObservedRunningTime="2026-07-02 19:09:04.093389818 +0000 UTC m=+51.870407386" Jul 02 19:09:04.807188 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.807160 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:04.847242 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.847218 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-rc8xt"] Jul 02 19:09:04.849825 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.849808 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rc8xt" Jul 02 19:09:04.853789 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.853769 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 02 19:09:04.853856 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.853791 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 02 19:09:04.853907 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.853770 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 02 19:09:04.853965 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.853769 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 02 19:09:04.854028 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.853770 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 02 19:09:04.854061 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.853770 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-h25kb\"" Jul 02 19:09:04.859351 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.859334 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-rc8xt"] Jul 02 19:09:04.894680 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.894655 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8148a379-8c15-4b5c-9a32-ad0d081a7205-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-rc8xt\" (UID: \"8148a379-8c15-4b5c-9a32-ad0d081a7205\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rc8xt" Jul 02 19:09:04.894778 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.894719 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8148a379-8c15-4b5c-9a32-ad0d081a7205-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-rc8xt\" (UID: \"8148a379-8c15-4b5c-9a32-ad0d081a7205\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rc8xt" Jul 02 19:09:04.894778 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.894752 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttnvd\" (UniqueName: \"kubernetes.io/projected/8148a379-8c15-4b5c-9a32-ad0d081a7205-kube-api-access-ttnvd\") pod \"prometheus-operator-567f7b8b7c-rc8xt\" (UID: \"8148a379-8c15-4b5c-9a32-ad0d081a7205\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rc8xt" Jul 02 19:09:04.894878 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.894779 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8148a379-8c15-4b5c-9a32-ad0d081a7205-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-rc8xt\" (UID: \"8148a379-8c15-4b5c-9a32-ad0d081a7205\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rc8xt" Jul 02 19:09:04.895486 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.895467 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:04.995511 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.995481 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8148a379-8c15-4b5c-9a32-ad0d081a7205-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-rc8xt\" (UID: \"8148a379-8c15-4b5c-9a32-ad0d081a7205\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rc8xt" Jul 02 19:09:04.995637 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.995520 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttnvd\" (UniqueName: \"kubernetes.io/projected/8148a379-8c15-4b5c-9a32-ad0d081a7205-kube-api-access-ttnvd\") pod \"prometheus-operator-567f7b8b7c-rc8xt\" (UID: \"8148a379-8c15-4b5c-9a32-ad0d081a7205\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rc8xt" Jul 02 19:09:04.995637 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.995571 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8148a379-8c15-4b5c-9a32-ad0d081a7205-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-rc8xt\" (UID: \"8148a379-8c15-4b5c-9a32-ad0d081a7205\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rc8xt" Jul 02 19:09:04.995730 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.995635 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8148a379-8c15-4b5c-9a32-ad0d081a7205-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-rc8xt\" (UID: \"8148a379-8c15-4b5c-9a32-ad0d081a7205\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rc8xt" Jul 02 19:09:04.995768 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:09:04.995743 2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 02 19:09:04.995829 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:09:04.995817 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8148a379-8c15-4b5c-9a32-ad0d081a7205-prometheus-operator-tls podName:8148a379-8c15-4b5c-9a32-ad0d081a7205 nodeName:}" failed. No retries permitted until 2026-07-02 19:09:05.495797032 +0000 UTC m=+53.272814581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/8148a379-8c15-4b5c-9a32-ad0d081a7205-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-rc8xt" (UID: "8148a379-8c15-4b5c-9a32-ad0d081a7205") : secret "prometheus-operator-tls" not found Jul 02 19:09:04.996230 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.996211 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8148a379-8c15-4b5c-9a32-ad0d081a7205-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-rc8xt\" (UID: \"8148a379-8c15-4b5c-9a32-ad0d081a7205\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rc8xt" Jul 02 19:09:04.999742 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:04.999721 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8148a379-8c15-4b5c-9a32-ad0d081a7205-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-rc8xt\" (UID: \"8148a379-8c15-4b5c-9a32-ad0d081a7205\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rc8xt" Jul 02 19:09:05.005710 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:05.005574 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttnvd\" (UniqueName: \"kubernetes.io/projected/8148a379-8c15-4b5c-9a32-ad0d081a7205-kube-api-access-ttnvd\") pod \"prometheus-operator-567f7b8b7c-rc8xt\" (UID: \"8148a379-8c15-4b5c-9a32-ad0d081a7205\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rc8xt" Jul 02 19:09:05.045309 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:05.044953 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v7lc8" event={"ID":"0499397a-c856-416b-b3e2-3b3998080ea7","Type":"ContainerStarted","Data":"64ec4183a746a3f73accc8b66a7437348fe768664e796ffdd8b1599a0c71e4b6"} Jul 02 19:09:05.500049 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:05.500017 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8148a379-8c15-4b5c-9a32-ad0d081a7205-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-rc8xt\" (UID: \"8148a379-8c15-4b5c-9a32-ad0d081a7205\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rc8xt" Jul 02 19:09:05.502880 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:05.502853 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8148a379-8c15-4b5c-9a32-ad0d081a7205-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-rc8xt\" (UID: \"8148a379-8c15-4b5c-9a32-ad0d081a7205\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rc8xt" Jul 02 19:09:05.533335 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:05.533304 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:05.759980 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:05.759902 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rc8xt" Jul 02 19:09:05.907742 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:05.907527 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-rc8xt"] Jul 02 19:09:05.911111 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:09:05.911074 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8148a379_8c15_4b5c_9a32_ad0d081a7205.slice/crio-f43b4d684f4f3861da08a85c62a3d44282ee1991f861dff9f08bf351d2b0efa9 WatchSource:0}: Error finding container f43b4d684f4f3861da08a85c62a3d44282ee1991f861dff9f08bf351d2b0efa9: Status 404 returned error can't find the container with id f43b4d684f4f3861da08a85c62a3d44282ee1991f861dff9f08bf351d2b0efa9 Jul 02 19:09:06.050393 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:06.050356 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rc8xt" event={"ID":"8148a379-8c15-4b5c-9a32-ad0d081a7205","Type":"ContainerStarted","Data":"f43b4d684f4f3861da08a85c62a3d44282ee1991f861dff9f08bf351d2b0efa9"} Jul 02 19:09:06.166431 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:06.166396 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:09:06.166620 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:06.166456 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:09:06.167820 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:06.167796 2562 patch_prober.go:28] interesting pod/console-846c85d97c-qpn8m container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" start-of-body= Jul 02 19:09:06.167931 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:06.167840 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-846c85d97c-qpn8m" podUID="d4fa7416-7a4f-4371-a663-46861e8e84e8" containerName="console" probeResult="failure" output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" Jul 02 19:09:07.914778 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:07.914739 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:08.019717 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:08.019685 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:08.058910 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:08.058878 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-v7lc8" event={"ID":"0499397a-c856-416b-b3e2-3b3998080ea7","Type":"ContainerStarted","Data":"40be199346c779ea72a807acd639d789cb78d1f4b6ff7c4a4a005ebc99072919"} Jul 02 19:09:08.082297 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:08.082253 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-v7lc8" podStartSLOduration=2.39788719 podStartE2EDuration="14.082238315s" totalImagesPullingTime="11.684351125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:08:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:09:08.080635244 +0000 UTC m=+55.857652831" watchObservedRunningTime="2026-07-02 19:09:08.082238315 +0000 UTC m=+55.859255876" Jul 02 19:09:08.358238 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:08.358206 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:09.064036 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:09.064001 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rc8xt" event={"ID":"8148a379-8c15-4b5c-9a32-ad0d081a7205","Type":"ContainerStarted","Data":"fe6426392bbbfea908da113011b0448affc18caffc12a1dab8fbd71effd12c94"} Jul 02 19:09:09.064426 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:09.064060 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rc8xt" event={"ID":"8148a379-8c15-4b5c-9a32-ad0d081a7205","Type":"ContainerStarted","Data":"bfdce0ac4c1624f57af875194cb007b7f264f1d72c7b67cd6ccfde315297d60f"} Jul 02 19:09:09.081218 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:09.081171 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rc8xt" podStartSLOduration=2.903696685 podStartE2EDuration="5.081156595s" totalImagesPullingTime="2.17745991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:09:09.080427867 +0000 UTC m=+56.857445437" watchObservedRunningTime="2026-07-02 19:09:09.081156595 +0000 UTC m=+56.858174162" Jul 02 19:09:09.246858 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:09.246823 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:09.306538 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:09.306507 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:09.996438 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:09.996409 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:10.163323 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:10.163291 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:10.463821 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:10.463780 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:10.528027 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:10.528002 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:11.088608 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.088549 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:11.321989 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.321959 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:11.358640 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.358575 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-rb49r"] Jul 02 19:09:11.361480 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.361466 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rb49r" Jul 02 19:09:11.364396 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.364376 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 02 19:09:11.365041 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.365027 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 02 19:09:11.365179 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.365164 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-l8s5b\"" Jul 02 19:09:11.376660 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.376635 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-rb49r"] Jul 02 19:09:11.391896 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.391878 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j"] Jul 02 19:09:11.394540 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.394526 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" Jul 02 19:09:11.397244 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.397225 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 02 19:09:11.397359 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.397269 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 02 19:09:11.397359 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.397322 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 02 19:09:11.397359 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.397334 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-m98fm\"" Jul 02 19:09:11.406717 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.406699 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m8g2g"] Jul 02 19:09:11.408827 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.408814 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.410806 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.410788 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j"] Jul 02 19:09:11.411916 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.411896 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 02 19:09:11.412620 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.412581 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 02 19:09:11.412922 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.412909 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 02 19:09:11.413089 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.413072 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jgkx2\"" Jul 02 19:09:11.438664 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.438642 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:11.442304 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.442285 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bfe4789-0421-4ae4-b909-10670eadfecd-sys\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.442377 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.442315 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km9c9\" (UniqueName: \"kubernetes.io/projected/788118e6-7cf7-4c96-aa08-7bebc980f993-kube-api-access-km9c9\") pod \"openshift-state-metrics-54cf96dfcf-rb49r\" (UID: \"788118e6-7cf7-4c96-aa08-7bebc980f993\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rb49r" Jul 02 19:09:11.442377 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.442332 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3338032-5243-4ed5-8ed8-6ad89c8b9a55-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-jgh6j\" (UID: \"a3338032-5243-4ed5-8ed8-6ad89c8b9a55\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" Jul 02 19:09:11.442455 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.442409 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5bfe4789-0421-4ae4-b909-10670eadfecd-node-exporter-textfile\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.442455 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.442438 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bfe4789-0421-4ae4-b909-10670eadfecd-metrics-client-ca\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.442518 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.442485 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5bfe4789-0421-4ae4-b909-10670eadfecd-node-exporter-wtmp\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.442518 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.442503 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5bfe4789-0421-4ae4-b909-10670eadfecd-root\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.442581 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.442525 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5bfe4789-0421-4ae4-b909-10670eadfecd-node-exporter-tls\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.442581 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.442556 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bfe4789-0421-4ae4-b909-10670eadfecd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.442581 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.442573 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/788118e6-7cf7-4c96-aa08-7bebc980f993-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-rb49r\" (UID: \"788118e6-7cf7-4c96-aa08-7bebc980f993\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rb49r" Jul 02 19:09:11.442704 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.442625 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3338032-5243-4ed5-8ed8-6ad89c8b9a55-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-jgh6j\" (UID: \"a3338032-5243-4ed5-8ed8-6ad89c8b9a55\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" Jul 02 19:09:11.442704 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.442644 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a3338032-5243-4ed5-8ed8-6ad89c8b9a55-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-jgh6j\" (UID: \"a3338032-5243-4ed5-8ed8-6ad89c8b9a55\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" Jul 02 19:09:11.442704 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.442661 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l67c\" (UniqueName: \"kubernetes.io/projected/5bfe4789-0421-4ae4-b909-10670eadfecd-kube-api-access-4l67c\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.442704 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.442685 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/788118e6-7cf7-4c96-aa08-7bebc980f993-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-rb49r\" (UID: \"788118e6-7cf7-4c96-aa08-7bebc980f993\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rb49r" Jul 02 19:09:11.442842 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.442708 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a3338032-5243-4ed5-8ed8-6ad89c8b9a55-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-jgh6j\" (UID: \"a3338032-5243-4ed5-8ed8-6ad89c8b9a55\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" Jul 02 19:09:11.442842 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.442752 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3338032-5243-4ed5-8ed8-6ad89c8b9a55-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-jgh6j\" (UID: \"a3338032-5243-4ed5-8ed8-6ad89c8b9a55\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" Jul 02 19:09:11.442842 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.442798 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/788118e6-7cf7-4c96-aa08-7bebc980f993-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-rb49r\" (UID: \"788118e6-7cf7-4c96-aa08-7bebc980f993\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rb49r" Jul 02 19:09:11.442842 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.442821 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5bfe4789-0421-4ae4-b909-10670eadfecd-node-exporter-accelerators-collector-config\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.442842 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.442835 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hm2m\" (UniqueName: \"kubernetes.io/projected/a3338032-5243-4ed5-8ed8-6ad89c8b9a55-kube-api-access-9hm2m\") pod \"kube-state-metrics-7c9dbb8b7d-jgh6j\" (UID: \"a3338032-5243-4ed5-8ed8-6ad89c8b9a55\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" Jul 02 19:09:11.525920 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.525895 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:11.544025 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544000 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bfe4789-0421-4ae4-b909-10670eadfecd-sys\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.544124 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544029 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km9c9\" (UniqueName: \"kubernetes.io/projected/788118e6-7cf7-4c96-aa08-7bebc980f993-kube-api-access-km9c9\") pod \"openshift-state-metrics-54cf96dfcf-rb49r\" (UID: \"788118e6-7cf7-4c96-aa08-7bebc980f993\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rb49r" Jul 02 19:09:11.544124 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544046 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3338032-5243-4ed5-8ed8-6ad89c8b9a55-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-jgh6j\" (UID: \"a3338032-5243-4ed5-8ed8-6ad89c8b9a55\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" Jul 02 19:09:11.544124 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544112 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bfe4789-0421-4ae4-b909-10670eadfecd-sys\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.544229 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544152 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5bfe4789-0421-4ae4-b909-10670eadfecd-node-exporter-textfile\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.544229 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544175 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bfe4789-0421-4ae4-b909-10670eadfecd-metrics-client-ca\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.544229 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544207 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5bfe4789-0421-4ae4-b909-10670eadfecd-node-exporter-wtmp\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.544363 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544250 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5bfe4789-0421-4ae4-b909-10670eadfecd-root\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.544363 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544273 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5bfe4789-0421-4ae4-b909-10670eadfecd-node-exporter-tls\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.544363 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544301 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bfe4789-0421-4ae4-b909-10670eadfecd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.544363 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544334 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/788118e6-7cf7-4c96-aa08-7bebc980f993-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-rb49r\" (UID: \"788118e6-7cf7-4c96-aa08-7bebc980f993\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rb49r" Jul 02 19:09:11.544363 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544334 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5bfe4789-0421-4ae4-b909-10670eadfecd-root\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.544622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544366 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3338032-5243-4ed5-8ed8-6ad89c8b9a55-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-jgh6j\" (UID: \"a3338032-5243-4ed5-8ed8-6ad89c8b9a55\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" Jul 02 19:09:11.544622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544390 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5bfe4789-0421-4ae4-b909-10670eadfecd-node-exporter-wtmp\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.544622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544394 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a3338032-5243-4ed5-8ed8-6ad89c8b9a55-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-jgh6j\" (UID: \"a3338032-5243-4ed5-8ed8-6ad89c8b9a55\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" Jul 02 19:09:11.544622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544456 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l67c\" (UniqueName: \"kubernetes.io/projected/5bfe4789-0421-4ae4-b909-10670eadfecd-kube-api-access-4l67c\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.544622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544466 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5bfe4789-0421-4ae4-b909-10670eadfecd-node-exporter-textfile\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.544622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544489 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/788118e6-7cf7-4c96-aa08-7bebc980f993-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-rb49r\" (UID: \"788118e6-7cf7-4c96-aa08-7bebc980f993\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rb49r" Jul 02 19:09:11.544622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544531 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a3338032-5243-4ed5-8ed8-6ad89c8b9a55-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-jgh6j\" (UID: \"a3338032-5243-4ed5-8ed8-6ad89c8b9a55\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" Jul 02 19:09:11.544622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544559 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3338032-5243-4ed5-8ed8-6ad89c8b9a55-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-jgh6j\" (UID: \"a3338032-5243-4ed5-8ed8-6ad89c8b9a55\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" Jul 02 19:09:11.545012 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544644 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/788118e6-7cf7-4c96-aa08-7bebc980f993-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-rb49r\" (UID: \"788118e6-7cf7-4c96-aa08-7bebc980f993\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rb49r" Jul 02 19:09:11.545012 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544677 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5bfe4789-0421-4ae4-b909-10670eadfecd-node-exporter-accelerators-collector-config\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.545012 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544720 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hm2m\" (UniqueName: \"kubernetes.io/projected/a3338032-5243-4ed5-8ed8-6ad89c8b9a55-kube-api-access-9hm2m\") pod \"kube-state-metrics-7c9dbb8b7d-jgh6j\" (UID: \"a3338032-5243-4ed5-8ed8-6ad89c8b9a55\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" Jul 02 19:09:11.545012 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544848 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bfe4789-0421-4ae4-b909-10670eadfecd-metrics-client-ca\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.545012 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.544848 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3338032-5243-4ed5-8ed8-6ad89c8b9a55-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-jgh6j\" (UID: \"a3338032-5243-4ed5-8ed8-6ad89c8b9a55\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" Jul 02 19:09:11.545220 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.545135 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a3338032-5243-4ed5-8ed8-6ad89c8b9a55-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-jgh6j\" (UID: \"a3338032-5243-4ed5-8ed8-6ad89c8b9a55\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" Jul 02 19:09:11.545628 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.545368 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/788118e6-7cf7-4c96-aa08-7bebc980f993-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-rb49r\" (UID: \"788118e6-7cf7-4c96-aa08-7bebc980f993\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rb49r" Jul 02 19:09:11.545628 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.545481 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a3338032-5243-4ed5-8ed8-6ad89c8b9a55-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-jgh6j\" (UID: \"a3338032-5243-4ed5-8ed8-6ad89c8b9a55\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" Jul 02 19:09:11.545841 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.545816 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5bfe4789-0421-4ae4-b909-10670eadfecd-node-exporter-accelerators-collector-config\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.548264 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.548235 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5bfe4789-0421-4ae4-b909-10670eadfecd-node-exporter-tls\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.548264 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.548254 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3338032-5243-4ed5-8ed8-6ad89c8b9a55-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-jgh6j\" (UID: \"a3338032-5243-4ed5-8ed8-6ad89c8b9a55\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" Jul 02 19:09:11.548421 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.548376 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bfe4789-0421-4ae4-b909-10670eadfecd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.548479 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.548421 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/788118e6-7cf7-4c96-aa08-7bebc980f993-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-rb49r\" (UID: \"788118e6-7cf7-4c96-aa08-7bebc980f993\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rb49r" Jul 02 19:09:11.548479 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.548458 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/788118e6-7cf7-4c96-aa08-7bebc980f993-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-rb49r\" (UID: \"788118e6-7cf7-4c96-aa08-7bebc980f993\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rb49r" Jul 02 19:09:11.548579 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.548561 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a3338032-5243-4ed5-8ed8-6ad89c8b9a55-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-jgh6j\" (UID: \"a3338032-5243-4ed5-8ed8-6ad89c8b9a55\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" Jul 02 19:09:11.552467 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.552440 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km9c9\" (UniqueName: \"kubernetes.io/projected/788118e6-7cf7-4c96-aa08-7bebc980f993-kube-api-access-km9c9\") pod \"openshift-state-metrics-54cf96dfcf-rb49r\" (UID: \"788118e6-7cf7-4c96-aa08-7bebc980f993\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rb49r" Jul 02 19:09:11.552548 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.552468 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hm2m\" (UniqueName: \"kubernetes.io/projected/a3338032-5243-4ed5-8ed8-6ad89c8b9a55-kube-api-access-9hm2m\") pod \"kube-state-metrics-7c9dbb8b7d-jgh6j\" (UID: \"a3338032-5243-4ed5-8ed8-6ad89c8b9a55\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" Jul 02 19:09:11.552666 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.552588 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l67c\" (UniqueName: \"kubernetes.io/projected/5bfe4789-0421-4ae4-b909-10670eadfecd-kube-api-access-4l67c\") pod \"node-exporter-m8g2g\" (UID: \"5bfe4789-0421-4ae4-b909-10670eadfecd\") " pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.626192 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.626127 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:11.670329 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.670308 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rb49r" Jul 02 19:09:11.706466 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.706438 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" Jul 02 19:09:11.718512 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.717739 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m8g2g" Jul 02 19:09:11.727110 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.727082 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:11.731033 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:09:11.730987 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfe4789_0421_4ae4_b909_10670eadfecd.slice/crio-9a07d6ff79fc195cc088ac8cea083ef2505f3e99e1ec0b6d822fb815d4b8f437 WatchSource:0}: Error finding container 9a07d6ff79fc195cc088ac8cea083ef2505f3e99e1ec0b6d822fb815d4b8f437: Status 404 returned error can't find the container with id 9a07d6ff79fc195cc088ac8cea083ef2505f3e99e1ec0b6d822fb815d4b8f437 Jul 02 19:09:11.806147 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.806109 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-rb49r"] Jul 02 19:09:11.808886 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:09:11.808859 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788118e6_7cf7_4c96_aa08_7bebc980f993.slice/crio-be7da621327b3117e6c14ffe87d30c5b710253a9ef55551c768fba362bc2dbc1 WatchSource:0}: Error finding container be7da621327b3117e6c14ffe87d30c5b710253a9ef55551c768fba362bc2dbc1: Status 404 returned error can't find the container with id be7da621327b3117e6c14ffe87d30c5b710253a9ef55551c768fba362bc2dbc1 Jul 02 19:09:11.846789 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:11.846766 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j"] Jul 02 19:09:11.849639 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:09:11.849612 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3338032_5243_4ed5_8ed8_6ad89c8b9a55.slice/crio-261bb8ddd8f4a15589f15c0e4b4c3f7fffda1aeeaa8145b08f87cf86adcb2c9a WatchSource:0}: Error finding container 261bb8ddd8f4a15589f15c0e4b4c3f7fffda1aeeaa8145b08f87cf86adcb2c9a: Status 404 returned error can't find the container with id 261bb8ddd8f4a15589f15c0e4b4c3f7fffda1aeeaa8145b08f87cf86adcb2c9a Jul 02 19:09:12.069911 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.069885 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:12.075019 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.074994 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rb49r" event={"ID":"788118e6-7cf7-4c96-aa08-7bebc980f993","Type":"ContainerStarted","Data":"33de879dc590a4e22abb3bc6e3b295f56725a8608b96c8eb371e6d98369e186e"} Jul 02 19:09:12.075142 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.075054 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rb49r" event={"ID":"788118e6-7cf7-4c96-aa08-7bebc980f993","Type":"ContainerStarted","Data":"f40bb54ab181138b385fc2f527bb756e09e9682c0579e357586ce1bb46a89f9c"} Jul 02 19:09:12.075142 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.075083 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rb49r" event={"ID":"788118e6-7cf7-4c96-aa08-7bebc980f993","Type":"ContainerStarted","Data":"be7da621327b3117e6c14ffe87d30c5b710253a9ef55551c768fba362bc2dbc1"} Jul 02 19:09:12.076113 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.076089 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m8g2g" event={"ID":"5bfe4789-0421-4ae4-b909-10670eadfecd","Type":"ContainerStarted","Data":"9a07d6ff79fc195cc088ac8cea083ef2505f3e99e1ec0b6d822fb815d4b8f437"} Jul 02 19:09:12.077197 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.077174 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" event={"ID":"a3338032-5243-4ed5-8ed8-6ad89c8b9a55","Type":"ContainerStarted","Data":"261bb8ddd8f4a15589f15c0e4b4c3f7fffda1aeeaa8145b08f87cf86adcb2c9a"} Jul 02 19:09:12.236799 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.236712 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:12.310556 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.310525 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:12.314292 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.314262 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 19:09:12.316872 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.316854 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.319538 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.319519 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 02 19:09:12.319767 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.319514 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 02 19:09:12.319767 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.319565 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 02 19:09:12.319767 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.319582 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 02 19:09:12.319767 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.319652 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 02 19:09:12.319919 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.319881 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 02 19:09:12.320133 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.320114 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 02 19:09:12.320247 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.320172 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 02 19:09:12.320312 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.320293 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 02 19:09:12.320418 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.320403 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bwcs5\"" Jul 02 19:09:12.333745 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.333725 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 19:09:12.351233 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.351209 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d3d6abf7-f154-43b1-8980-f06175e47018-config-volume\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.351343 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.351272 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d3d6abf7-f154-43b1-8980-f06175e47018-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.351343 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.351289 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3d6abf7-f154-43b1-8980-f06175e47018-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.351428 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.351352 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3d6abf7-f154-43b1-8980-f06175e47018-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.351428 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.351411 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d3d6abf7-f154-43b1-8980-f06175e47018-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.351504 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.351433 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3d6abf7-f154-43b1-8980-f06175e47018-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.351504 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.351468 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d3d6abf7-f154-43b1-8980-f06175e47018-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.351504 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.351499 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3d6abf7-f154-43b1-8980-f06175e47018-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.351635 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.351517 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3d6abf7-f154-43b1-8980-f06175e47018-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.351635 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.351537 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3d6abf7-f154-43b1-8980-f06175e47018-web-config\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.351635 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.351554 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3d6abf7-f154-43b1-8980-f06175e47018-config-out\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.351635 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.351570 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d3d6abf7-f154-43b1-8980-f06175e47018-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.351774 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.351650 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kbn4\" (UniqueName: \"kubernetes.io/projected/d3d6abf7-f154-43b1-8980-f06175e47018-kube-api-access-6kbn4\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.452219 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.452185 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d3d6abf7-f154-43b1-8980-f06175e47018-config-volume\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.452396 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.452274 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d3d6abf7-f154-43b1-8980-f06175e47018-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.452396 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.452303 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3d6abf7-f154-43b1-8980-f06175e47018-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.452396 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.452333 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3d6abf7-f154-43b1-8980-f06175e47018-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.452396 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.452363 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d3d6abf7-f154-43b1-8980-f06175e47018-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.452396 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.452393 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3d6abf7-f154-43b1-8980-f06175e47018-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.452843 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.452462 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d3d6abf7-f154-43b1-8980-f06175e47018-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.452843 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.452525 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3d6abf7-f154-43b1-8980-f06175e47018-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.452843 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.452558 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3d6abf7-f154-43b1-8980-f06175e47018-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.452843 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.452630 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3d6abf7-f154-43b1-8980-f06175e47018-web-config\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.452843 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.452753 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d3d6abf7-f154-43b1-8980-f06175e47018-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.453136 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.452897 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3d6abf7-f154-43b1-8980-f06175e47018-config-out\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.453136 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.452933 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d3d6abf7-f154-43b1-8980-f06175e47018-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.453136 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.452965 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kbn4\" (UniqueName: \"kubernetes.io/projected/d3d6abf7-f154-43b1-8980-f06175e47018-kube-api-access-6kbn4\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.453727 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.453383 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3d6abf7-f154-43b1-8980-f06175e47018-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.453727 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.453490 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3d6abf7-f154-43b1-8980-f06175e47018-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.456121 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.456074 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d3d6abf7-f154-43b1-8980-f06175e47018-config-volume\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.456584 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.456560 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3d6abf7-f154-43b1-8980-f06175e47018-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.456704 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.456685 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3d6abf7-f154-43b1-8980-f06175e47018-config-out\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.456752 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.456707 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d3d6abf7-f154-43b1-8980-f06175e47018-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.456890 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.456842 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3d6abf7-f154-43b1-8980-f06175e47018-web-config\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.457171 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.457120 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3d6abf7-f154-43b1-8980-f06175e47018-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.457932 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.457912 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3d6abf7-f154-43b1-8980-f06175e47018-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.458804 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.458781 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d3d6abf7-f154-43b1-8980-f06175e47018-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.458894 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.458783 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d3d6abf7-f154-43b1-8980-f06175e47018-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.461166 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.461145 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kbn4\" (UniqueName: \"kubernetes.io/projected/d3d6abf7-f154-43b1-8980-f06175e47018-kube-api-access-6kbn4\") pod \"alertmanager-main-0\" (UID: \"d3d6abf7-f154-43b1-8980-f06175e47018\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.534909 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.534876 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:12.630878 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.630850 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bwcs5\"" Jul 02 19:09:12.638454 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.638429 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:09:12.802694 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.802611 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 19:09:12.806188 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:09:12.806160 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d6abf7_f154_43b1_8980_f06175e47018.slice/crio-1d62f41eb0048454312a7f5c480b54327853df1b1ab5e70be54d5745ea591292 WatchSource:0}: Error finding container 1d62f41eb0048454312a7f5c480b54327853df1b1ab5e70be54d5745ea591292: Status 404 returned error can't find the container with id 1d62f41eb0048454312a7f5c480b54327853df1b1ab5e70be54d5745ea591292 Jul 02 19:09:12.947976 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:12.947950 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xm2wv" Jul 02 19:09:13.082429 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:13.082336 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3d6abf7-f154-43b1-8980-f06175e47018","Type":"ContainerStarted","Data":"1d62f41eb0048454312a7f5c480b54327853df1b1ab5e70be54d5745ea591292"} Jul 02 19:09:13.082429 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:13.082371 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:13.180354 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:13.180321 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:14.047374 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.047329 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9pdxg" Jul 02 19:09:14.070573 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.070517 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:14.088562 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.088532 2562 generic.go:356] "Generic (PLEG): container finished" podID="5bfe4789-0421-4ae4-b909-10670eadfecd" containerID="78c7e63d362117fc7e0989cfbd12307e1009298e400b0202397e4a688e3b5a2a" exitCode=0 Jul 02 19:09:14.088761 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.088689 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m8g2g" event={"ID":"5bfe4789-0421-4ae4-b909-10670eadfecd","Type":"ContainerDied","Data":"78c7e63d362117fc7e0989cfbd12307e1009298e400b0202397e4a688e3b5a2a"} Jul 02 19:09:14.093857 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.093715 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" event={"ID":"a3338032-5243-4ed5-8ed8-6ad89c8b9a55","Type":"ContainerStarted","Data":"0022c241c9c2ff3643ec5de425e967bda819a2148cc3d637ff96974a2cf5c2de"} Jul 02 19:09:14.093857 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.093786 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" event={"ID":"a3338032-5243-4ed5-8ed8-6ad89c8b9a55","Type":"ContainerStarted","Data":"5cd05ecc8c4555a0a2a1a601047929df79af4b6c606db0cf7d8aa104cce264d5"} Jul 02 19:09:14.093857 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.093819 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" event={"ID":"a3338032-5243-4ed5-8ed8-6ad89c8b9a55","Type":"ContainerStarted","Data":"9bdc1d244724b3e9ec5e43d4e9fac0731acdd9ec58c010888de19500f5601fc7"} Jul 02 19:09:14.097080 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.097028 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rb49r" event={"ID":"788118e6-7cf7-4c96-aa08-7bebc980f993","Type":"ContainerStarted","Data":"e5a3fc0f0310dbc7107cc66cc17109138fdc41317fe8d77a0a153d790551e76d"} Jul 02 19:09:14.130043 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.129995 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rb49r" podStartSLOduration=1.448415641 podStartE2EDuration="3.129979706s" totalImagesPullingTime="1.681564065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:09:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:09:14.128805782 +0000 UTC m=+61.905823370" watchObservedRunningTime="2026-07-02 19:09:14.129979706 +0000 UTC m=+61.906997272" Jul 02 19:09:14.152476 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.152418 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgh6j" podStartSLOduration=1.4008093449999999 podStartE2EDuration="3.152397389s" totalImagesPullingTime="1.751588044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:09:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:09:14.150953185 +0000 UTC m=+61.927970752" watchObservedRunningTime="2026-07-02 19:09:14.152397389 +0000 UTC m=+61.929414961" Jul 02 19:09:14.156245 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.156222 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:14.226133 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.226089 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:14.330904 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.330866 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:14.365428 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.365364 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-9d645bff7-k9gt7"] Jul 02 19:09:14.371400 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.371378 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.374108 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.374087 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 02 19:09:14.374418 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.374241 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 02 19:09:14.374418 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.374246 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fcme2cks48pjr\"" Jul 02 19:09:14.374555 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.374524 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 02 19:09:14.374648 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.374564 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tjfdr\"" Jul 02 19:09:14.374648 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.374616 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 02 19:09:14.374870 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.374855 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 02 19:09:14.385274 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.385235 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-9d645bff7-k9gt7"] Jul 02 19:09:14.415037 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.415014 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:14.469416 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.469333 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dedb7476-dbe0-4bde-89a6-391c39a4057e-secret-grpc-tls\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.469479 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.469416 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dedb7476-dbe0-4bde-89a6-391c39a4057e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.469479 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.469470 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdhll\" (UniqueName: \"kubernetes.io/projected/dedb7476-dbe0-4bde-89a6-391c39a4057e-kube-api-access-cdhll\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.469616 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.469528 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dedb7476-dbe0-4bde-89a6-391c39a4057e-metrics-client-ca\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.469616 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.469559 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dedb7476-dbe0-4bde-89a6-391c39a4057e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.469732 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.469650 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dedb7476-dbe0-4bde-89a6-391c39a4057e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.469732 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.469686 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dedb7476-dbe0-4bde-89a6-391c39a4057e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.469732 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.469724 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dedb7476-dbe0-4bde-89a6-391c39a4057e-secret-thanos-querier-tls\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.562915 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.562887 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:14.570847 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.570826 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dedb7476-dbe0-4bde-89a6-391c39a4057e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.570933 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.570872 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdhll\" (UniqueName: \"kubernetes.io/projected/dedb7476-dbe0-4bde-89a6-391c39a4057e-kube-api-access-cdhll\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.570933 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.570914 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dedb7476-dbe0-4bde-89a6-391c39a4057e-metrics-client-ca\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.571013 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.570947 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dedb7476-dbe0-4bde-89a6-391c39a4057e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.571013 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.571006 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dedb7476-dbe0-4bde-89a6-391c39a4057e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.571088 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.571035 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dedb7476-dbe0-4bde-89a6-391c39a4057e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.571088 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.571072 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dedb7476-dbe0-4bde-89a6-391c39a4057e-secret-thanos-querier-tls\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.571177 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.571096 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dedb7476-dbe0-4bde-89a6-391c39a4057e-secret-grpc-tls\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.571788 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.571764 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dedb7476-dbe0-4bde-89a6-391c39a4057e-metrics-client-ca\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.574088 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.574054 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dedb7476-dbe0-4bde-89a6-391c39a4057e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.574205 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.574154 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/dedb7476-dbe0-4bde-89a6-391c39a4057e-secret-thanos-querier-tls\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.574205 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.574173 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dedb7476-dbe0-4bde-89a6-391c39a4057e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.574305 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.574216 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/dedb7476-dbe0-4bde-89a6-391c39a4057e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.574305 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.574235 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/dedb7476-dbe0-4bde-89a6-391c39a4057e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.574305 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.574275 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dedb7476-dbe0-4bde-89a6-391c39a4057e-secret-grpc-tls\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.578522 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.578504 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdhll\" (UniqueName: \"kubernetes.io/projected/dedb7476-dbe0-4bde-89a6-391c39a4057e-kube-api-access-cdhll\") pod \"thanos-querier-9d645bff7-k9gt7\" (UID: \"dedb7476-dbe0-4bde-89a6-391c39a4057e\") " pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.681885 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.681832 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:14.762653 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.762563 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:14.866039 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:14.866010 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-9d645bff7-k9gt7"] Jul 02 19:09:14.869108 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:09:14.869085 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddedb7476_dbe0_4bde_89a6_391c39a4057e.slice/crio-3c65743496357025acaf3aef12f9dd3a2df11423c870ad5a4ccd58751f9779f4 WatchSource:0}: Error finding container 3c65743496357025acaf3aef12f9dd3a2df11423c870ad5a4ccd58751f9779f4: Status 404 returned error can't find the container with id 3c65743496357025acaf3aef12f9dd3a2df11423c870ad5a4ccd58751f9779f4 Jul 02 19:09:15.102911 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.102875 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m8g2g" event={"ID":"5bfe4789-0421-4ae4-b909-10670eadfecd","Type":"ContainerStarted","Data":"9032499a4bd2a58f5927e144e53db8f971456ec0c9df714d1cc269063db52bd4"} Jul 02 19:09:15.103356 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.102936 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m8g2g" event={"ID":"5bfe4789-0421-4ae4-b909-10670eadfecd","Type":"ContainerStarted","Data":"0b4db6f5b16e101ed59952bba3828ceb1dbc092b64cb111b43681edc2a5d90cf"} Jul 02 19:09:15.104342 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.104320 2562 generic.go:356] "Generic (PLEG): container finished" podID="d3d6abf7-f154-43b1-8980-f06175e47018" containerID="c1852255abea4d4375c7209666ba5037a24546f3994d5f6d2a19374d73595080" exitCode=0 Jul 02 19:09:15.104420 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.104381 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3d6abf7-f154-43b1-8980-f06175e47018","Type":"ContainerDied","Data":"c1852255abea4d4375c7209666ba5037a24546f3994d5f6d2a19374d73595080"} Jul 02 19:09:15.105562 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.105527 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" event={"ID":"dedb7476-dbe0-4bde-89a6-391c39a4057e","Type":"ContainerStarted","Data":"3c65743496357025acaf3aef12f9dd3a2df11423c870ad5a4ccd58751f9779f4"} Jul 02 19:09:15.126423 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.126379 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m8g2g" podStartSLOduration=2.151283325 podStartE2EDuration="4.126365486s" totalImagesPullingTime="1.866418554s" totalInitContainerRuntime="108.663607ms" isStatefulPod=true podCreationTimestamp="2026-07-02 19:09:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:09:15.125505905 +0000 UTC m=+62.902523497" watchObservedRunningTime="2026-07-02 19:09:15.126365486 +0000 UTC m=+62.903383053" Jul 02 19:09:15.585742 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.585714 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:15.666712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.666682 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-555f6bb4cf-s5gfc"] Jul 02 19:09:15.686029 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.686006 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-555f6bb4cf-s5gfc"] Jul 02 19:09:15.686175 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.686094 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:15.688734 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.688712 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 02 19:09:15.688867 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.688857 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zzdgq\"" Jul 02 19:09:15.688925 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.688896 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 02 19:09:15.688982 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.688915 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 02 19:09:15.688982 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.688974 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 02 19:09:15.689069 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.688918 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2tjci374p2s9\"" Jul 02 19:09:15.782754 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.782727 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/95367456-3d2d-465e-b1a9-59c6f8fc66a1-audit-log\") pod \"metrics-server-555f6bb4cf-s5gfc\" (UID: \"95367456-3d2d-465e-b1a9-59c6f8fc66a1\") " pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:15.782754 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.782760 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fr6f\" (UniqueName: \"kubernetes.io/projected/95367456-3d2d-465e-b1a9-59c6f8fc66a1-kube-api-access-8fr6f\") pod \"metrics-server-555f6bb4cf-s5gfc\" (UID: \"95367456-3d2d-465e-b1a9-59c6f8fc66a1\") " pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:15.782965 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.782792 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95367456-3d2d-465e-b1a9-59c6f8fc66a1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-555f6bb4cf-s5gfc\" (UID: \"95367456-3d2d-465e-b1a9-59c6f8fc66a1\") " pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:15.782965 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.782866 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/95367456-3d2d-465e-b1a9-59c6f8fc66a1-metrics-server-audit-profiles\") pod \"metrics-server-555f6bb4cf-s5gfc\" (UID: \"95367456-3d2d-465e-b1a9-59c6f8fc66a1\") " pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:15.783075 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.782956 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/95367456-3d2d-465e-b1a9-59c6f8fc66a1-client-ca-bundle\") pod \"metrics-server-555f6bb4cf-s5gfc\" (UID: \"95367456-3d2d-465e-b1a9-59c6f8fc66a1\") " pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:15.783075 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.783000 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/95367456-3d2d-465e-b1a9-59c6f8fc66a1-secret-metrics-server-client-certs\") pod \"metrics-server-555f6bb4cf-s5gfc\" (UID: \"95367456-3d2d-465e-b1a9-59c6f8fc66a1\") " pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:15.783160 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.783077 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/95367456-3d2d-465e-b1a9-59c6f8fc66a1-secret-metrics-server-tls\") pod \"metrics-server-555f6bb4cf-s5gfc\" (UID: \"95367456-3d2d-465e-b1a9-59c6f8fc66a1\") " pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:15.884191 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.884101 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fr6f\" (UniqueName: \"kubernetes.io/projected/95367456-3d2d-465e-b1a9-59c6f8fc66a1-kube-api-access-8fr6f\") pod \"metrics-server-555f6bb4cf-s5gfc\" (UID: \"95367456-3d2d-465e-b1a9-59c6f8fc66a1\") " pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:15.884191 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.884164 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95367456-3d2d-465e-b1a9-59c6f8fc66a1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-555f6bb4cf-s5gfc\" (UID: \"95367456-3d2d-465e-b1a9-59c6f8fc66a1\") " pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:15.884410 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.884242 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/95367456-3d2d-465e-b1a9-59c6f8fc66a1-metrics-server-audit-profiles\") pod \"metrics-server-555f6bb4cf-s5gfc\" (UID: \"95367456-3d2d-465e-b1a9-59c6f8fc66a1\") " pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:15.884410 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.884276 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/95367456-3d2d-465e-b1a9-59c6f8fc66a1-client-ca-bundle\") pod \"metrics-server-555f6bb4cf-s5gfc\" (UID: \"95367456-3d2d-465e-b1a9-59c6f8fc66a1\") " pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:15.884410 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.884344 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/95367456-3d2d-465e-b1a9-59c6f8fc66a1-secret-metrics-server-client-certs\") pod \"metrics-server-555f6bb4cf-s5gfc\" (UID: \"95367456-3d2d-465e-b1a9-59c6f8fc66a1\") " pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:15.884410 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.884404 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/95367456-3d2d-465e-b1a9-59c6f8fc66a1-secret-metrics-server-tls\") pod \"metrics-server-555f6bb4cf-s5gfc\" (UID: \"95367456-3d2d-465e-b1a9-59c6f8fc66a1\") " pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:15.884623 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.884461 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/95367456-3d2d-465e-b1a9-59c6f8fc66a1-audit-log\") pod \"metrics-server-555f6bb4cf-s5gfc\" (UID: \"95367456-3d2d-465e-b1a9-59c6f8fc66a1\") " pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:15.884918 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.884890 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/95367456-3d2d-465e-b1a9-59c6f8fc66a1-audit-log\") pod \"metrics-server-555f6bb4cf-s5gfc\" (UID: \"95367456-3d2d-465e-b1a9-59c6f8fc66a1\") " pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:15.885029 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.884989 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95367456-3d2d-465e-b1a9-59c6f8fc66a1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-555f6bb4cf-s5gfc\" (UID: \"95367456-3d2d-465e-b1a9-59c6f8fc66a1\") " pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:15.885275 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.885217 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/95367456-3d2d-465e-b1a9-59c6f8fc66a1-metrics-server-audit-profiles\") pod \"metrics-server-555f6bb4cf-s5gfc\" (UID: \"95367456-3d2d-465e-b1a9-59c6f8fc66a1\") " pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:15.887442 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.887421 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/95367456-3d2d-465e-b1a9-59c6f8fc66a1-secret-metrics-server-client-certs\") pod \"metrics-server-555f6bb4cf-s5gfc\" (UID: \"95367456-3d2d-465e-b1a9-59c6f8fc66a1\") " pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:15.887536 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.887420 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/95367456-3d2d-465e-b1a9-59c6f8fc66a1-client-ca-bundle\") pod \"metrics-server-555f6bb4cf-s5gfc\" (UID: \"95367456-3d2d-465e-b1a9-59c6f8fc66a1\") " pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:15.887536 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.887450 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/95367456-3d2d-465e-b1a9-59c6f8fc66a1-secret-metrics-server-tls\") pod \"metrics-server-555f6bb4cf-s5gfc\" (UID: \"95367456-3d2d-465e-b1a9-59c6f8fc66a1\") " pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:15.892548 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.892525 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fr6f\" (UniqueName: \"kubernetes.io/projected/95367456-3d2d-465e-b1a9-59c6f8fc66a1-kube-api-access-8fr6f\") pod \"metrics-server-555f6bb4cf-s5gfc\" (UID: \"95367456-3d2d-465e-b1a9-59c6f8fc66a1\") " pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:15.996155 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:15.996125 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:16.023230 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.023195 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:16.032889 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.032863 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-vbd2x"] Jul 02 19:09:16.036709 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.036685 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vbd2x" Jul 02 19:09:16.039675 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.039651 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rlz5r\"" Jul 02 19:09:16.039890 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.039872 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 02 19:09:16.048090 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.047859 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-vbd2x"] Jul 02 19:09:16.083691 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.083576 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:16.087940 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.087909 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjpl4\" (UniqueName: \"kubernetes.io/projected/26588082-05d8-446e-ad43-96ce0e8e5d96-kube-api-access-rjpl4\") pod \"monitoring-plugin-d5dcff7cf-vbd2x\" (UID: \"26588082-05d8-446e-ad43-96ce0e8e5d96\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vbd2x" Jul 02 19:09:16.088024 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.087962 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26588082-05d8-446e-ad43-96ce0e8e5d96-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-vbd2x\" (UID: \"26588082-05d8-446e-ad43-96ce0e8e5d96\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vbd2x" Jul 02 19:09:16.143314 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.143227 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:16.167552 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.167517 2562 patch_prober.go:28] interesting pod/console-846c85d97c-qpn8m container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" start-of-body= Jul 02 19:09:16.167713 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.167575 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-846c85d97c-qpn8m" podUID="d4fa7416-7a4f-4371-a663-46861e8e84e8" containerName="console" probeResult="failure" output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" Jul 02 19:09:16.172262 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.172233 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-555f6bb4cf-s5gfc"] Jul 02 19:09:16.174772 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:09:16.174746 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95367456_3d2d_465e_b1a9_59c6f8fc66a1.slice/crio-31f1d86580d8704bb841d939acd41d1604448c45133c13f4d95a5a58edc8fdfd WatchSource:0}: Error finding container 31f1d86580d8704bb841d939acd41d1604448c45133c13f4d95a5a58edc8fdfd: Status 404 returned error can't find the container with id 31f1d86580d8704bb841d939acd41d1604448c45133c13f4d95a5a58edc8fdfd Jul 02 19:09:16.188659 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.188566 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjpl4\" (UniqueName: \"kubernetes.io/projected/26588082-05d8-446e-ad43-96ce0e8e5d96-kube-api-access-rjpl4\") pod \"monitoring-plugin-d5dcff7cf-vbd2x\" (UID: \"26588082-05d8-446e-ad43-96ce0e8e5d96\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vbd2x" Jul 02 19:09:16.188659 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.188644 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26588082-05d8-446e-ad43-96ce0e8e5d96-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-vbd2x\" (UID: \"26588082-05d8-446e-ad43-96ce0e8e5d96\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vbd2x" Jul 02 19:09:16.191507 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.191483 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/26588082-05d8-446e-ad43-96ce0e8e5d96-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-vbd2x\" (UID: \"26588082-05d8-446e-ad43-96ce0e8e5d96\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vbd2x" Jul 02 19:09:16.197084 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.197052 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjpl4\" (UniqueName: \"kubernetes.io/projected/26588082-05d8-446e-ad43-96ce0e8e5d96-kube-api-access-rjpl4\") pod \"monitoring-plugin-d5dcff7cf-vbd2x\" (UID: \"26588082-05d8-446e-ad43-96ce0e8e5d96\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vbd2x" Jul 02 19:09:16.358812 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.358774 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vbd2x" Jul 02 19:09:16.409376 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.409260 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:16.410187 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.410147 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-846c85d97c-qpn8m"] Jul 02 19:09:16.438653 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.437804 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-748c864687-drtmf"] Jul 02 19:09:16.441703 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.441681 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:16.446215 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.446186 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-748c864687-drtmf"] Jul 02 19:09:16.484410 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.484378 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:16.491228 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.491205 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-console-serving-cert\") pod \"console-748c864687-drtmf\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:16.491374 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.491247 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-console-oauth-config\") pod \"console-748c864687-drtmf\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:16.491374 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.491329 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-service-ca\") pod \"console-748c864687-drtmf\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:16.491374 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.491362 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-oauth-serving-cert\") pod \"console-748c864687-drtmf\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:16.491559 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.491391 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-console-config\") pod \"console-748c864687-drtmf\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:16.491559 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.491417 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-trusted-ca-bundle\") pod \"console-748c864687-drtmf\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:16.491559 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.491529 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh46f\" (UniqueName: \"kubernetes.io/projected/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-kube-api-access-fh46f\") pod \"console-748c864687-drtmf\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:16.519174 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.519118 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-vbd2x"] Jul 02 19:09:16.521779 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:09:16.521749 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26588082_05d8_446e_ad43_96ce0e8e5d96.slice/crio-5218873a3b81974704ce5c5c2a19fd4de8f1b01bf10db725baff92c391872a87 WatchSource:0}: Error finding container 5218873a3b81974704ce5c5c2a19fd4de8f1b01bf10db725baff92c391872a87: Status 404 returned error can't find the container with id 5218873a3b81974704ce5c5c2a19fd4de8f1b01bf10db725baff92c391872a87 Jul 02 19:09:16.592274 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.592248 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh46f\" (UniqueName: \"kubernetes.io/projected/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-kube-api-access-fh46f\") pod \"console-748c864687-drtmf\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:16.592417 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.592317 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-console-serving-cert\") pod \"console-748c864687-drtmf\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:16.592417 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.592364 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-console-oauth-config\") pod \"console-748c864687-drtmf\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:16.592417 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.592402 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-service-ca\") pod \"console-748c864687-drtmf\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:16.592576 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.592426 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-oauth-serving-cert\") pod \"console-748c864687-drtmf\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:16.592576 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.592452 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-console-config\") pod \"console-748c864687-drtmf\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:16.592576 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.592477 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-trusted-ca-bundle\") pod \"console-748c864687-drtmf\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:16.593332 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.593280 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-service-ca\") pod \"console-748c864687-drtmf\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:16.593332 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.593321 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-console-config\") pod \"console-748c864687-drtmf\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:16.593903 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.593857 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-trusted-ca-bundle\") pod \"console-748c864687-drtmf\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:16.594043 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.594021 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-oauth-serving-cert\") pod \"console-748c864687-drtmf\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:16.595039 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.595022 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-console-serving-cert\") pod \"console-748c864687-drtmf\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:16.595221 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.595205 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-console-oauth-config\") pod \"console-748c864687-drtmf\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:16.600864 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.600848 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh46f\" (UniqueName: \"kubernetes.io/projected/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-kube-api-access-fh46f\") pod \"console-748c864687-drtmf\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:16.756573 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:16.756404 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:17.116819 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.116778 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vbd2x" event={"ID":"26588082-05d8-446e-ad43-96ce0e8e5d96","Type":"ContainerStarted","Data":"5218873a3b81974704ce5c5c2a19fd4de8f1b01bf10db725baff92c391872a87"} Jul 02 19:09:17.118078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.118041 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" event={"ID":"95367456-3d2d-465e-b1a9-59c6f8fc66a1","Type":"ContainerStarted","Data":"31f1d86580d8704bb841d939acd41d1604448c45133c13f4d95a5a58edc8fdfd"} Jul 02 19:09:17.294838 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.294807 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:17.411063 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.411012 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:17.417879 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.417814 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-748c864687-drtmf"] Jul 02 19:09:17.679640 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.678067 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:17.679640 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.678243 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 19:09:17.683668 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.682566 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.687320 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.687297 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1mahtdsjb5je7\"" Jul 02 19:09:17.688388 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.688363 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 02 19:09:17.689362 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.689175 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 02 19:09:17.689362 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.689289 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 02 19:09:17.690009 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.689626 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 02 19:09:17.690009 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.689689 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tjpq8\"" Jul 02 19:09:17.690009 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.689792 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 02 19:09:17.690009 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.689843 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 02 19:09:17.692219 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.691112 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 02 19:09:17.692219 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.691543 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 02 19:09:17.692219 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.691756 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 02 19:09:17.693720 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.693207 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 02 19:09:17.693720 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.693258 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 02 19:09:17.693720 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.693391 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 02 19:09:17.699316 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.698169 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 02 19:09:17.708013 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.701373 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 02 19:09:17.713505 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.713353 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 19:09:17.809845 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.808214 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.809845 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.808267 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.809845 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.809068 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca978e38-b363-422c-8c68-ea06ad2b98c1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.809845 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.809122 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca978e38-b363-422c-8c68-ea06ad2b98c1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.809845 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.809155 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca978e38-b363-422c-8c68-ea06ad2b98c1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.809845 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.809183 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.809845 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.809220 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca978e38-b363-422c-8c68-ea06ad2b98c1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.809845 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.809272 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-config\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.809845 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.809319 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca978e38-b363-422c-8c68-ea06ad2b98c1-config-out\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.809845 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.809351 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ca978e38-b363-422c-8c68-ea06ad2b98c1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.809845 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.809377 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.809845 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.809401 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ca978e38-b363-422c-8c68-ea06ad2b98c1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.809845 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.809438 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ca978e38-b363-422c-8c68-ea06ad2b98c1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.809845 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.809470 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-web-config\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.809845 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.809493 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.809845 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.809521 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.810681 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.809560 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ca978e38-b363-422c-8c68-ea06ad2b98c1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.810681 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.809584 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca978e38-b363-422c-8c68-ea06ad2b98c1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.810681 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.809638 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.810681 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.809661 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58vkz\" (UniqueName: \"kubernetes.io/projected/ca978e38-b363-422c-8c68-ea06ad2b98c1-kube-api-access-58vkz\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.914622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.913343 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ca978e38-b363-422c-8c68-ea06ad2b98c1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.914622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.913403 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-web-config\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.914622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.913430 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.914622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.913460 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.914622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.913497 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ca978e38-b363-422c-8c68-ea06ad2b98c1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.914622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.913519 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca978e38-b363-422c-8c68-ea06ad2b98c1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.914622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.913538 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.914622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.913552 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58vkz\" (UniqueName: \"kubernetes.io/projected/ca978e38-b363-422c-8c68-ea06ad2b98c1-kube-api-access-58vkz\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.914622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.913577 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.914622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.913618 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.914622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.913646 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca978e38-b363-422c-8c68-ea06ad2b98c1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.914622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.913677 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca978e38-b363-422c-8c68-ea06ad2b98c1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.914622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.913703 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca978e38-b363-422c-8c68-ea06ad2b98c1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.914622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.913729 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.914622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.913755 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca978e38-b363-422c-8c68-ea06ad2b98c1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.914622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.913796 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-config\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.914622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.913838 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca978e38-b363-422c-8c68-ea06ad2b98c1-config-out\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.915728 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.913863 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ca978e38-b363-422c-8c68-ea06ad2b98c1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.915728 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.913888 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.915728 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.913908 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ca978e38-b363-422c-8c68-ea06ad2b98c1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.915728 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.914503 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ca978e38-b363-422c-8c68-ea06ad2b98c1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.918232 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.917491 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca978e38-b363-422c-8c68-ea06ad2b98c1-config-out\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.918232 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.917561 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca978e38-b363-422c-8c68-ea06ad2b98c1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.919828 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.919803 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca978e38-b363-422c-8c68-ea06ad2b98c1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.920032 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.920012 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.920362 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.920339 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-config\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.920515 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.920491 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca978e38-b363-422c-8c68-ea06ad2b98c1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.920957 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.920935 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ca978e38-b363-422c-8c68-ea06ad2b98c1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.921216 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.921192 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ca978e38-b363-422c-8c68-ea06ad2b98c1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.922128 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.922094 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca978e38-b363-422c-8c68-ea06ad2b98c1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.922246 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.922218 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.922828 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.922314 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.922828 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.922688 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-web-config\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.923000 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.922836 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ca978e38-b363-422c-8c68-ea06ad2b98c1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.923000 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.922967 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca978e38-b363-422c-8c68-ea06ad2b98c1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.924855 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.923918 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.924855 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.924778 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.925104 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.925074 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.925535 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.925512 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca978e38-b363-422c-8c68-ea06ad2b98c1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.933138 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.933078 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58vkz\" (UniqueName: \"kubernetes.io/projected/ca978e38-b363-422c-8c68-ea06ad2b98c1-kube-api-access-58vkz\") pod \"prometheus-k8s-0\" (UID: \"ca978e38-b363-422c-8c68-ea06ad2b98c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:17.997208 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:17.996821 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:18.129376 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.127679 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-748c864687-drtmf" event={"ID":"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb","Type":"ContainerStarted","Data":"ffa3eebb2acbe89e4609679e96db1c0e36d192f18c6bbf2c1349ea1daa839e95"} Jul 02 19:09:18.129376 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.127742 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-748c864687-drtmf" event={"ID":"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb","Type":"ContainerStarted","Data":"57698e89b769efe27fb756218be96a7a66d88bc813fdea7cbc256f30cfaf5db2"} Jul 02 19:09:18.151368 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.149053 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-748c864687-drtmf" podStartSLOduration=2.14903001 podStartE2EDuration="2.14903001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:09:18.147253585 +0000 UTC m=+65.924271185" watchObservedRunningTime="2026-07-02 19:09:18.14903001 +0000 UTC m=+65.926047577" Jul 02 19:09:18.166955 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.164066 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" event={"ID":"dedb7476-dbe0-4bde-89a6-391c39a4057e","Type":"ContainerStarted","Data":"8cc0fea19cad28f77d4fd55df5a78ea1c9afbcee96b7adcea5892abc70feff04"} Jul 02 19:09:18.166955 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.164148 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" event={"ID":"dedb7476-dbe0-4bde-89a6-391c39a4057e","Type":"ContainerStarted","Data":"de83b476a3cec792a6b376e7d8d708b8518b02d1b07b1178b2faabd850c749d4"} Jul 02 19:09:18.222085 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.222055 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 19:09:18.418641 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.418581 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs\") pod \"network-metrics-daemon-6z7f2\" (UID: \"6a167d1c-5370-45bb-96c3-9deb3c9ddb50\") " pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:09:18.421533 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.421512 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 19:09:18.431411 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.431388 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a167d1c-5370-45bb-96c3-9deb3c9ddb50-metrics-certs\") pod \"network-metrics-daemon-6z7f2\" (UID: \"6a167d1c-5370-45bb-96c3-9deb3c9ddb50\") " pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:09:18.519622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.519555 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret\") pod \"global-pull-secret-syncer-hqszg\" (UID: \"448316b6-cd2d-4c73-8ca6-103e4c64f18e\") " pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:09:18.522970 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.522946 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 19:09:18.532750 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.532726 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/448316b6-cd2d-4c73-8ca6-103e4c64f18e-original-pull-secret\") pod \"global-pull-secret-syncer-hqszg\" (UID: \"448316b6-cd2d-4c73-8ca6-103e4c64f18e\") " pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:09:18.560772 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:09:18.560747 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca978e38_b363_422c_8c68_ea06ad2b98c1.slice/crio-ad2a62a3591eb4b678817ed5d480183d9658afa66f08f56c73e9f051f74a3174 WatchSource:0}: Error finding container ad2a62a3591eb4b678817ed5d480183d9658afa66f08f56c73e9f051f74a3174: Status 404 returned error can't find the container with id ad2a62a3591eb4b678817ed5d480183d9658afa66f08f56c73e9f051f74a3174 Jul 02 19:09:18.602609 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.602578 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r7xgd\"" Jul 02 19:09:18.610714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.610694 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6z7f2" Jul 02 19:09:18.620216 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.620193 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnwvq\" (UniqueName: \"kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq\") pod \"network-check-target-mhpt4\" (UID: \"b552dfdb-6037-4463-a58d-59a748531ee1\") " pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:09:18.623344 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.623322 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 19:09:18.633136 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.633117 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 19:09:18.643759 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.643736 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnwvq\" (UniqueName: \"kubernetes.io/projected/b552dfdb-6037-4463-a58d-59a748531ee1-kube-api-access-hnwvq\") pod \"network-check-target-mhpt4\" (UID: \"b552dfdb-6037-4463-a58d-59a748531ee1\") " pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:09:18.653976 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.653959 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5rf5h\"" Jul 02 19:09:18.659046 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.659028 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hqszg" Jul 02 19:09:18.662403 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.662385 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:09:18.832250 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.832172 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:18.886300 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:18.886257 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:19.171169 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:19.171105 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vbd2x" event={"ID":"26588082-05d8-446e-ad43-96ce0e8e5d96","Type":"ContainerStarted","Data":"0da30d2ebe563fc68e103cb9486a18fb3eca33dc4c4f8578952dd7c121d3ac74"} Jul 02 19:09:19.172146 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:19.172088 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vbd2x" Jul 02 19:09:19.174839 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:19.174297 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" event={"ID":"95367456-3d2d-465e-b1a9-59c6f8fc66a1","Type":"ContainerStarted","Data":"c3d1df7ba9286012b42afa6b3ff5d6028af07d989c1b80b31d419d8565c12591"} Jul 02 19:09:19.178376 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:19.178355 2562 generic.go:356] "Generic (PLEG): container finished" podID="ca978e38-b363-422c-8c68-ea06ad2b98c1" containerID="6ad4b911f163e43a8615d99692d4052d9b2daadb770331bd04872fcfd02fa655" exitCode=0 Jul 02 19:09:19.178664 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:19.178522 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca978e38-b363-422c-8c68-ea06ad2b98c1","Type":"ContainerDied","Data":"6ad4b911f163e43a8615d99692d4052d9b2daadb770331bd04872fcfd02fa655"} Jul 02 19:09:19.178664 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:19.178612 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca978e38-b363-422c-8c68-ea06ad2b98c1","Type":"ContainerStarted","Data":"ad2a62a3591eb4b678817ed5d480183d9658afa66f08f56c73e9f051f74a3174"} Jul 02 19:09:19.179687 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:19.179555 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vbd2x" Jul 02 19:09:19.183221 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:19.183203 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3d6abf7-f154-43b1-8980-f06175e47018","Type":"ContainerStarted","Data":"be09adc54d0b98248b1742b7385284c8316801d76f5747983e3c5f567d08e93d"} Jul 02 19:09:19.183388 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:19.183370 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3d6abf7-f154-43b1-8980-f06175e47018","Type":"ContainerStarted","Data":"2e8f7f6f0e7b1d438c01b329a4e80d8c4ece1c7bec6f6b284904e272169ebea2"} Jul 02 19:09:19.183455 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:19.183442 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3d6abf7-f154-43b1-8980-f06175e47018","Type":"ContainerStarted","Data":"2c9b0e6e42152d3ff6cdc627ee814c95f86592fdc1e593ffb8cf688ed46f253f"} Jul 02 19:09:19.187283 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:19.187190 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" event={"ID":"dedb7476-dbe0-4bde-89a6-391c39a4057e","Type":"ContainerStarted","Data":"1cfa6bd17e55644f00afd3b9ffc608953843a226b1e2ac7b69cd6c3d873cc88c"} Jul 02 19:09:19.189907 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:19.189888 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6z7f2"] Jul 02 19:09:19.200408 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:19.200256 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-vbd2x" podStartSLOduration=0.743469291 podStartE2EDuration="3.200237582s" totalImagesPullingTime="2.456768291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:09:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:09:19.198638435 +0000 UTC m=+66.975656019" watchObservedRunningTime="2026-07-02 19:09:19.200237582 +0000 UTC m=+66.977255144" Jul 02 19:09:19.200944 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:09:19.200925 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a167d1c_5370_45bb_96c3_9deb3c9ddb50.slice/crio-4be96fb6f33a67c1e139f23a789db0ee34410e4918132b0422fd4d88ad91bc1e WatchSource:0}: Error finding container 4be96fb6f33a67c1e139f23a789db0ee34410e4918132b0422fd4d88ad91bc1e: Status 404 returned error can't find the container with id 4be96fb6f33a67c1e139f23a789db0ee34410e4918132b0422fd4d88ad91bc1e Jul 02 19:09:19.224365 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:19.224339 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mhpt4"] Jul 02 19:09:19.237318 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:09:19.236646 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb552dfdb_6037_4463_a58d_59a748531ee1.slice/crio-fe282ce40d2801398e8f85eb1b1e8fa388b46afe553d3a75767e608f1ac83de9 WatchSource:0}: Error finding container fe282ce40d2801398e8f85eb1b1e8fa388b46afe553d3a75767e608f1ac83de9: Status 404 returned error can't find the container with id fe282ce40d2801398e8f85eb1b1e8fa388b46afe553d3a75767e608f1ac83de9 Jul 02 19:09:19.241286 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:19.241217 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hqszg"] Jul 02 19:09:19.243061 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:19.243013 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" podStartSLOduration=1.437373771 podStartE2EDuration="4.242983605s" totalImagesPullingTime="2.805609834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:09:19.236845938 +0000 UTC m=+67.013863519" watchObservedRunningTime="2026-07-02 19:09:19.242983605 +0000 UTC m=+67.020001173" Jul 02 19:09:20.195668 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:20.195614 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6z7f2" event={"ID":"6a167d1c-5370-45bb-96c3-9deb3c9ddb50","Type":"ContainerStarted","Data":"4be96fb6f33a67c1e139f23a789db0ee34410e4918132b0422fd4d88ad91bc1e"} Jul 02 19:09:20.201841 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:20.200511 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" event={"ID":"dedb7476-dbe0-4bde-89a6-391c39a4057e","Type":"ContainerStarted","Data":"77cba7c322917f01301bba8f81c125e6d2b6d323a554740291af2f90fd59aab1"} Jul 02 19:09:20.201841 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:20.200586 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" event={"ID":"dedb7476-dbe0-4bde-89a6-391c39a4057e","Type":"ContainerStarted","Data":"104ce6c28cbd059dbe35e53678cd60f33f931d6feac19a619c63fd303e8c6edc"} Jul 02 19:09:20.201841 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:20.200729 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" event={"ID":"dedb7476-dbe0-4bde-89a6-391c39a4057e","Type":"ContainerStarted","Data":"12d1599a0f377407c9fb9328d8b0bef55d7e2d810c570a18386547b10b17fa57"} Jul 02 19:09:20.201841 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:20.201776 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:20.203821 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:20.203763 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mhpt4" event={"ID":"b552dfdb-6037-4463-a58d-59a748531ee1","Type":"ContainerStarted","Data":"fe282ce40d2801398e8f85eb1b1e8fa388b46afe553d3a75767e608f1ac83de9"} Jul 02 19:09:20.211222 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:20.210414 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3d6abf7-f154-43b1-8980-f06175e47018","Type":"ContainerStarted","Data":"a18483f92dfbada9144380cbbbbb35ff358587950ec6624ba555886679a0eda4"} Jul 02 19:09:20.211222 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:20.210459 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3d6abf7-f154-43b1-8980-f06175e47018","Type":"ContainerStarted","Data":"bfb69d1fce899be7936bc6c52ad3b331adea673b17d654ed6fe3a4a403220345"} Jul 02 19:09:20.211222 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:20.210486 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d3d6abf7-f154-43b1-8980-f06175e47018","Type":"ContainerStarted","Data":"9fe1017eef642c43fca6b61596ed9ff504ce0ad1623182ec5647aa49a3ce5beb"} Jul 02 19:09:20.217920 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:20.217894 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hqszg" event={"ID":"448316b6-cd2d-4c73-8ca6-103e4c64f18e","Type":"ContainerStarted","Data":"c27d431353fe3897a9fa37223da62a525d98ee78b0026edc93f35c7c75788bac"} Jul 02 19:09:20.256900 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:20.256824 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" podStartSLOduration=2.521828085 podStartE2EDuration="6.25680464s" totalImagesPullingTime="3.734976555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:09:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:09:20.22727389 +0000 UTC m=+68.004291458" watchObservedRunningTime="2026-07-02 19:09:20.25680464 +0000 UTC m=+68.033822212" Jul 02 19:09:20.257554 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:20.257515 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.353569843 podStartE2EDuration="8.257503994s" totalImagesPullingTime="4.880946294s" totalInitContainerRuntime="22.987857ms" isStatefulPod=false podCreationTimestamp="2026-07-02 19:09:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:09:20.25643818 +0000 UTC m=+68.033455773" watchObservedRunningTime="2026-07-02 19:09:20.257503994 +0000 UTC m=+68.034521561" Jul 02 19:09:21.225753 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:21.225717 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6z7f2" event={"ID":"6a167d1c-5370-45bb-96c3-9deb3c9ddb50","Type":"ContainerStarted","Data":"5b570365f69b547c95227ec1903b3fce110ce411d13a6b6826c3e0af98378a6d"} Jul 02 19:09:21.226217 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:21.225777 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6z7f2" event={"ID":"6a167d1c-5370-45bb-96c3-9deb3c9ddb50","Type":"ContainerStarted","Data":"fe4e5aa219d85152bacfd58d86bafaa8a07fc91f012d8a327d94241935bad759"} Jul 02 19:09:21.248238 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:21.248188 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6z7f2" podStartSLOduration=67.132220894 podStartE2EDuration="1m8.248172747s" totalImagesPullingTime="1.115951853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:09:21.24696457 +0000 UTC m=+69.023982170" watchObservedRunningTime="2026-07-02 19:09:21.248172747 +0000 UTC m=+69.025190316" Jul 02 19:09:21.361572 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:21.361535 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:21.435754 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:21.435665 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:24.209303 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:24.209266 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:24.309062 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:24.309035 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:24.409611 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:24.409544 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:26.249812 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:26.249723 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mhpt4" event={"ID":"b552dfdb-6037-4463-a58d-59a748531ee1","Type":"ContainerStarted","Data":"3d12773eb4d8b1b85987cc4fef52d49067aa2ae2020b5eddaf3c03a14812b706"} Jul 02 19:09:26.249812 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:26.249789 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:09:26.252994 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:26.252969 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca978e38-b363-422c-8c68-ea06ad2b98c1","Type":"ContainerStarted","Data":"cd97dee308c2f52c00883405fd414617c754dcc2039c4ea90a4f7ebe5815ef5e"} Jul 02 19:09:26.253542 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:26.253515 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca978e38-b363-422c-8c68-ea06ad2b98c1","Type":"ContainerStarted","Data":"90385a329227204c0032c4909ce28b3c32a1a6da46029d081747bb8c57c2e906"} Jul 02 19:09:26.253649 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:26.253625 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca978e38-b363-422c-8c68-ea06ad2b98c1","Type":"ContainerStarted","Data":"3f5bb3072a68ef6a4f79a7220ca30e8a59249361b0770638e111cea791447841"} Jul 02 19:09:26.257611 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:26.253759 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca978e38-b363-422c-8c68-ea06ad2b98c1","Type":"ContainerStarted","Data":"8dea4e4276d44e816dee9fffa371cc42aa59812b63b7c04918591e96e022fa8b"} Jul 02 19:09:26.257611 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:26.253832 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca978e38-b363-422c-8c68-ea06ad2b98c1","Type":"ContainerStarted","Data":"0065b88607282e2737a636a75160a2e26d1e622f1ef4a1e1eb567e6f944f5709"} Jul 02 19:09:26.257611 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:26.253890 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ca978e38-b363-422c-8c68-ea06ad2b98c1","Type":"ContainerStarted","Data":"8a8266249c3463e83f789257720e0ea62bdf2e5cce51d985f3d1297565268f17"} Jul 02 19:09:26.259310 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:26.259282 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hqszg" event={"ID":"448316b6-cd2d-4c73-8ca6-103e4c64f18e","Type":"ContainerStarted","Data":"c0699ef68b03c4e1ef3c40440ec8a44ddfd5fcf127a4e74ae86fd77501f7c051"} Jul 02 19:09:26.268030 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:26.267986 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mhpt4" podStartSLOduration=66.917843575 podStartE2EDuration="1m13.267973117s" totalImagesPullingTime="6.350129542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:09:26.266793789 +0000 UTC m=+74.043811369" watchObservedRunningTime="2026-07-02 19:09:26.267973117 +0000 UTC m=+74.044990685" Jul 02 19:09:26.282037 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:26.282000 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hqszg" podStartSLOduration=66.917630549 podStartE2EDuration="1m13.281988363s" totalImagesPullingTime="6.364357814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:09:26.281303437 +0000 UTC m=+74.058321005" watchObservedRunningTime="2026-07-02 19:09:26.281988363 +0000 UTC m=+74.059005930" Jul 02 19:09:26.310144 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:26.310067 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.839433127 podStartE2EDuration="9.310049792s" totalImagesPullingTime="6.429051395s" totalInitContainerRuntime="41.56527ms" isStatefulPod=false podCreationTimestamp="2026-07-02 19:09:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:09:26.308054364 +0000 UTC m=+74.085071968" watchObservedRunningTime="2026-07-02 19:09:26.310049792 +0000 UTC m=+74.087067362" Jul 02 19:09:26.757215 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:26.757178 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:26.757445 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:26.757237 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:26.761868 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:26.761848 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:27.237996 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:27.237974 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-9d645bff7-k9gt7" Jul 02 19:09:27.268884 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:27.268862 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-748c864687-drtmf" Jul 02 19:09:27.436676 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:27.436625 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:27.537166 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:27.537130 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:27.655824 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:27.655747 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:27.748231 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:27.748198 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:27.912377 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:27.912273 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:27.975652 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:27.975618 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:27.997567 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:27.997540 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:09:29.373956 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:29.373917 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:29.660178 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:29.660098 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:32.229538 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:32.229508 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:32.329127 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:32.329097 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:35.996613 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:35.996473 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:35.996613 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:35.996567 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:41.442006 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:41.441941 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-846c85d97c-qpn8m" podUID="d4fa7416-7a4f-4371-a663-46861e8e84e8" containerName="console" containerID="cri-o://873fa6fb0ec71e8670941d7b87ef0714158244b2393a6f78d7fe951b9f6d96ee" gracePeriod=15 Jul 02 19:09:42.221634 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.221582 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:09:42.326414 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.326343 2562 generic.go:356] "Generic (PLEG): container finished" podID="d4fa7416-7a4f-4371-a663-46861e8e84e8" containerID="873fa6fb0ec71e8670941d7b87ef0714158244b2393a6f78d7fe951b9f6d96ee" exitCode=0 Jul 02 19:09:42.326414 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.326391 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846c85d97c-qpn8m" Jul 02 19:09:42.326414 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.326399 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846c85d97c-qpn8m" event={"ID":"d4fa7416-7a4f-4371-a663-46861e8e84e8","Type":"ContainerDied","Data":"873fa6fb0ec71e8670941d7b87ef0714158244b2393a6f78d7fe951b9f6d96ee"} Jul 02 19:09:42.326618 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.326463 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846c85d97c-qpn8m" event={"ID":"d4fa7416-7a4f-4371-a663-46861e8e84e8","Type":"ContainerDied","Data":"2b29b979e85496b8d648f5c42085ad2265d17487e938cbe30c8f34aed63687bd"} Jul 02 19:09:42.326618 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.326484 2562 scope.go:117] "RemoveContainer" containerID="873fa6fb0ec71e8670941d7b87ef0714158244b2393a6f78d7fe951b9f6d96ee" Jul 02 19:09:42.335656 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.335636 2562 scope.go:117] "RemoveContainer" containerID="873fa6fb0ec71e8670941d7b87ef0714158244b2393a6f78d7fe951b9f6d96ee" Jul 02 19:09:42.335932 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:09:42.335910 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"873fa6fb0ec71e8670941d7b87ef0714158244b2393a6f78d7fe951b9f6d96ee\": container with ID starting with 873fa6fb0ec71e8670941d7b87ef0714158244b2393a6f78d7fe951b9f6d96ee not found: ID does not exist" containerID="873fa6fb0ec71e8670941d7b87ef0714158244b2393a6f78d7fe951b9f6d96ee" Jul 02 19:09:42.336011 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.335940 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873fa6fb0ec71e8670941d7b87ef0714158244b2393a6f78d7fe951b9f6d96ee"} err="failed to get container status \"873fa6fb0ec71e8670941d7b87ef0714158244b2393a6f78d7fe951b9f6d96ee\": rpc error: code = NotFound desc = could not find container \"873fa6fb0ec71e8670941d7b87ef0714158244b2393a6f78d7fe951b9f6d96ee\": container with ID starting with 873fa6fb0ec71e8670941d7b87ef0714158244b2393a6f78d7fe951b9f6d96ee not found: ID does not exist" Jul 02 19:09:42.345198 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.345173 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-oauth-serving-cert\") pod \"d4fa7416-7a4f-4371-a663-46861e8e84e8\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " Jul 02 19:09:42.345304 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.345258 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-console-config\") pod \"d4fa7416-7a4f-4371-a663-46861e8e84e8\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " Jul 02 19:09:42.345304 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.345279 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzvbk\" (UniqueName: \"kubernetes.io/projected/d4fa7416-7a4f-4371-a663-46861e8e84e8-kube-api-access-bzvbk\") pod \"d4fa7416-7a4f-4371-a663-46861e8e84e8\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " Jul 02 19:09:42.345304 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.345300 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-trusted-ca-bundle\") pod \"d4fa7416-7a4f-4371-a663-46861e8e84e8\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " Jul 02 19:09:42.345465 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.345318 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4fa7416-7a4f-4371-a663-46861e8e84e8-console-serving-cert\") pod \"d4fa7416-7a4f-4371-a663-46861e8e84e8\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " Jul 02 19:09:42.345465 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.345342 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4fa7416-7a4f-4371-a663-46861e8e84e8-console-oauth-config\") pod \"d4fa7416-7a4f-4371-a663-46861e8e84e8\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " Jul 02 19:09:42.345465 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.345378 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-service-ca\") pod \"d4fa7416-7a4f-4371-a663-46861e8e84e8\" (UID: \"d4fa7416-7a4f-4371-a663-46861e8e84e8\") " Jul 02 19:09:42.345716 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.345578 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d4fa7416-7a4f-4371-a663-46861e8e84e8" (UID: "d4fa7416-7a4f-4371-a663-46861e8e84e8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:09:42.345893 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.345720 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-console-config" (OuterVolumeSpecName: "console-config") pod "d4fa7416-7a4f-4371-a663-46861e8e84e8" (UID: "d4fa7416-7a4f-4371-a663-46861e8e84e8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:09:42.346018 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.345915 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-service-ca" (OuterVolumeSpecName: "service-ca") pod "d4fa7416-7a4f-4371-a663-46861e8e84e8" (UID: "d4fa7416-7a4f-4371-a663-46861e8e84e8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:09:42.346018 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.345929 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d4fa7416-7a4f-4371-a663-46861e8e84e8" (UID: "d4fa7416-7a4f-4371-a663-46861e8e84e8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:09:42.347782 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.347757 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fa7416-7a4f-4371-a663-46861e8e84e8-kube-api-access-bzvbk" (OuterVolumeSpecName: "kube-api-access-bzvbk") pod "d4fa7416-7a4f-4371-a663-46861e8e84e8" (UID: "d4fa7416-7a4f-4371-a663-46861e8e84e8"). InnerVolumeSpecName "kube-api-access-bzvbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:09:42.347865 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.347776 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4fa7416-7a4f-4371-a663-46861e8e84e8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d4fa7416-7a4f-4371-a663-46861e8e84e8" (UID: "d4fa7416-7a4f-4371-a663-46861e8e84e8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:09:42.347865 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.347826 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4fa7416-7a4f-4371-a663-46861e8e84e8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d4fa7416-7a4f-4371-a663-46861e8e84e8" (UID: "d4fa7416-7a4f-4371-a663-46861e8e84e8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:09:42.446761 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.446737 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4fa7416-7a4f-4371-a663-46861e8e84e8-console-oauth-config\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:09:42.446761 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.446759 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-service-ca\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:09:42.447082 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.446769 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-oauth-serving-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:09:42.447082 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.446777 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-console-config\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:09:42.447082 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.446786 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzvbk\" (UniqueName: \"kubernetes.io/projected/d4fa7416-7a4f-4371-a663-46861e8e84e8-kube-api-access-bzvbk\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:09:42.447082 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.446793 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4fa7416-7a4f-4371-a663-46861e8e84e8-trusted-ca-bundle\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:09:42.447082 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.446801 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4fa7416-7a4f-4371-a663-46861e8e84e8-console-serving-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:09:42.647136 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.647109 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-846c85d97c-qpn8m"] Jul 02 19:09:42.650960 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.650940 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-846c85d97c-qpn8m"] Jul 02 19:09:42.776055 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:42.776028 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4fa7416-7a4f-4371-a663-46861e8e84e8" path="/var/lib/kubelet/pods/d4fa7416-7a4f-4371-a663-46861e8e84e8/volumes" Jul 02 19:09:44.376086 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:44.376036 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:44.477804 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:44.477762 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:56.002010 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:56.001977 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:56.005827 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:56.005802 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-555f6bb4cf-s5gfc" Jul 02 19:09:57.267145 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:57.267117 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mhpt4" Jul 02 19:09:59.368712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:59.368677 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:59.422509 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:09:59.422448 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:09.812247 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:09.812216 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:09.873607 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:09.873529 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:17.997643 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:17.997583 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:10:18.018452 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:18.018419 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:10:18.493363 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:18.493334 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:10:30.702667 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:30.702546 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:30.986802 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:30.986715 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:35.805423 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:35.805385 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-587fc79689-mqtmt"] Jul 02 19:10:35.805910 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:35.805878 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4fa7416-7a4f-4371-a663-46861e8e84e8" containerName="console" Jul 02 19:10:35.805910 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:35.805897 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4fa7416-7a4f-4371-a663-46861e8e84e8" containerName="console" Jul 02 19:10:35.806030 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:35.805979 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4fa7416-7a4f-4371-a663-46861e8e84e8" containerName="console" Jul 02 19:10:35.808826 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:35.808804 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:35.811561 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:35.811529 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 02 19:10:35.812125 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:35.812108 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 02 19:10:35.812397 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:35.812380 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 02 19:10:35.812514 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:35.812489 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 02 19:10:35.812514 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:35.812443 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-s56rk\"" Jul 02 19:10:35.812770 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:35.812522 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 02 19:10:35.818964 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:35.818933 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 02 19:10:35.823066 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:35.823036 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-587fc79689-mqtmt"] Jul 02 19:10:35.967613 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:35.967562 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e86d2639-7d04-4eb8-8b5d-2b79101a066f-secret-telemeter-client\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:35.967613 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:35.967616 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e86d2639-7d04-4eb8-8b5d-2b79101a066f-telemeter-client-tls\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:35.967804 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:35.967640 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e86d2639-7d04-4eb8-8b5d-2b79101a066f-metrics-client-ca\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:35.967804 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:35.967667 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl5pp\" (UniqueName: \"kubernetes.io/projected/e86d2639-7d04-4eb8-8b5d-2b79101a066f-kube-api-access-tl5pp\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:35.967804 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:35.967707 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e86d2639-7d04-4eb8-8b5d-2b79101a066f-serving-certs-ca-bundle\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:35.967804 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:35.967762 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e86d2639-7d04-4eb8-8b5d-2b79101a066f-federate-client-tls\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:35.967804 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:35.967797 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e86d2639-7d04-4eb8-8b5d-2b79101a066f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:35.967957 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:35.967814 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e86d2639-7d04-4eb8-8b5d-2b79101a066f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:36.068419 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:36.068350 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e86d2639-7d04-4eb8-8b5d-2b79101a066f-secret-telemeter-client\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:36.068419 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:36.068383 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e86d2639-7d04-4eb8-8b5d-2b79101a066f-telemeter-client-tls\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:36.068419 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:36.068405 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e86d2639-7d04-4eb8-8b5d-2b79101a066f-metrics-client-ca\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:36.068643 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:36.068426 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl5pp\" (UniqueName: \"kubernetes.io/projected/e86d2639-7d04-4eb8-8b5d-2b79101a066f-kube-api-access-tl5pp\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:36.068643 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:36.068628 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e86d2639-7d04-4eb8-8b5d-2b79101a066f-serving-certs-ca-bundle\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:36.068720 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:36.068697 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e86d2639-7d04-4eb8-8b5d-2b79101a066f-federate-client-tls\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:36.068771 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:36.068752 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e86d2639-7d04-4eb8-8b5d-2b79101a066f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:36.068824 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:36.068786 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e86d2639-7d04-4eb8-8b5d-2b79101a066f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:36.069244 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:36.069218 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e86d2639-7d04-4eb8-8b5d-2b79101a066f-metrics-client-ca\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:36.069530 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:36.069509 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e86d2639-7d04-4eb8-8b5d-2b79101a066f-serving-certs-ca-bundle\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:36.069862 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:36.069839 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e86d2639-7d04-4eb8-8b5d-2b79101a066f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:36.071173 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:36.071130 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e86d2639-7d04-4eb8-8b5d-2b79101a066f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:36.071285 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:36.071265 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e86d2639-7d04-4eb8-8b5d-2b79101a066f-federate-client-tls\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:36.071533 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:36.071517 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e86d2639-7d04-4eb8-8b5d-2b79101a066f-telemeter-client-tls\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:36.071611 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:36.071574 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e86d2639-7d04-4eb8-8b5d-2b79101a066f-secret-telemeter-client\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:36.077621 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:36.077583 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl5pp\" (UniqueName: \"kubernetes.io/projected/e86d2639-7d04-4eb8-8b5d-2b79101a066f-kube-api-access-tl5pp\") pod \"telemeter-client-587fc79689-mqtmt\" (UID: \"e86d2639-7d04-4eb8-8b5d-2b79101a066f\") " pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:36.124235 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:36.124210 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" Jul 02 19:10:36.130404 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:36.130374 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:36.257480 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:36.257437 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-587fc79689-mqtmt"] Jul 02 19:10:36.261728 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:10:36.261698 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86d2639_7d04_4eb8_8b5d_2b79101a066f.slice/crio-b01ffd861b730a6e40f574437b17f8127a351b782fe1dbe9294f1b15af35a5da WatchSource:0}: Error finding container b01ffd861b730a6e40f574437b17f8127a351b782fe1dbe9294f1b15af35a5da: Status 404 returned error can't find the container with id b01ffd861b730a6e40f574437b17f8127a351b782fe1dbe9294f1b15af35a5da Jul 02 19:10:36.548490 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:36.548454 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" event={"ID":"e86d2639-7d04-4eb8-8b5d-2b79101a066f","Type":"ContainerStarted","Data":"b01ffd861b730a6e40f574437b17f8127a351b782fe1dbe9294f1b15af35a5da"} Jul 02 19:10:38.558416 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:38.558332 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" event={"ID":"e86d2639-7d04-4eb8-8b5d-2b79101a066f","Type":"ContainerStarted","Data":"c7ec9f316f28677e74c04f9209ce0a54c9a6fa0f9e560cb445e487e96820c461"} Jul 02 19:10:38.558416 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:38.558401 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" event={"ID":"e86d2639-7d04-4eb8-8b5d-2b79101a066f","Type":"ContainerStarted","Data":"b9d635b8c2b85ab1ef08d129da2ecab579c8735fce5ce0081b36b1a84e21290f"} Jul 02 19:10:38.558816 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:38.558424 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" event={"ID":"e86d2639-7d04-4eb8-8b5d-2b79101a066f","Type":"ContainerStarted","Data":"51a9acbe88b960fd38170f019855d8b79ffaf9af8c5b0fd49651ff172e8d00c5"} Jul 02 19:10:38.583445 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:38.583396 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-587fc79689-mqtmt" podStartSLOduration=1.5471253 podStartE2EDuration="3.583381724s" totalImagesPullingTime="2.036256424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:10:38.58039939 +0000 UTC m=+146.357416979" watchObservedRunningTime="2026-07-02 19:10:38.583381724 +0000 UTC m=+146.360399326" Jul 02 19:10:39.454138 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:39.454100 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:39.602849 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:39.602808 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:40.203944 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:40.203909 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:40.304276 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:40.304243 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:45.891067 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:45.891023 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:45.949224 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:45.949198 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:50.398386 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:50.398352 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-748c864687-drtmf"] Jul 02 19:10:50.522911 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:50.522877 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:50.856763 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:50.856731 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:57.917435 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:57.917402 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:58.090644 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:10:58.090613 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:11:00.030710 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:00.030682 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:11:01.613038 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:01.613002 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:11:01.712189 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:01.712157 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:11:01.785509 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:01.785478 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:11:14.393997 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:14.393911 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:11:14.493784 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:14.493724 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:11:15.419274 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.419233 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-748c864687-drtmf" podUID="a3db4cec-c581-4e0d-960a-dcf5bc1e35eb" containerName="console" containerID="cri-o://ffa3eebb2acbe89e4609679e96db1c0e36d192f18c6bbf2c1349ea1daa839e95" gracePeriod=15 Jul 02 19:11:15.658014 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.657993 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-748c864687-drtmf" Jul 02 19:11:15.713420 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.713336 2562 generic.go:356] "Generic (PLEG): container finished" podID="a3db4cec-c581-4e0d-960a-dcf5bc1e35eb" containerID="ffa3eebb2acbe89e4609679e96db1c0e36d192f18c6bbf2c1349ea1daa839e95" exitCode=0 Jul 02 19:11:15.713420 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.713397 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-748c864687-drtmf" Jul 02 19:11:15.713580 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.713416 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-748c864687-drtmf" event={"ID":"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb","Type":"ContainerDied","Data":"ffa3eebb2acbe89e4609679e96db1c0e36d192f18c6bbf2c1349ea1daa839e95"} Jul 02 19:11:15.713580 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.713515 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-748c864687-drtmf" event={"ID":"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb","Type":"ContainerDied","Data":"57698e89b769efe27fb756218be96a7a66d88bc813fdea7cbc256f30cfaf5db2"} Jul 02 19:11:15.713580 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.713541 2562 scope.go:117] "RemoveContainer" containerID="ffa3eebb2acbe89e4609679e96db1c0e36d192f18c6bbf2c1349ea1daa839e95" Jul 02 19:11:15.722179 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.722163 2562 scope.go:117] "RemoveContainer" containerID="ffa3eebb2acbe89e4609679e96db1c0e36d192f18c6bbf2c1349ea1daa839e95" Jul 02 19:11:15.722440 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:11:15.722425 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffa3eebb2acbe89e4609679e96db1c0e36d192f18c6bbf2c1349ea1daa839e95\": container with ID starting with ffa3eebb2acbe89e4609679e96db1c0e36d192f18c6bbf2c1349ea1daa839e95 not found: ID does not exist" containerID="ffa3eebb2acbe89e4609679e96db1c0e36d192f18c6bbf2c1349ea1daa839e95" Jul 02 19:11:15.722489 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.722449 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa3eebb2acbe89e4609679e96db1c0e36d192f18c6bbf2c1349ea1daa839e95"} err="failed to get container status \"ffa3eebb2acbe89e4609679e96db1c0e36d192f18c6bbf2c1349ea1daa839e95\": rpc error: code = NotFound desc = could not find container \"ffa3eebb2acbe89e4609679e96db1c0e36d192f18c6bbf2c1349ea1daa839e95\": container with ID starting with ffa3eebb2acbe89e4609679e96db1c0e36d192f18c6bbf2c1349ea1daa839e95 not found: ID does not exist" Jul 02 19:11:15.792585 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.792563 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-console-serving-cert\") pod \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " Jul 02 19:11:15.792727 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.792618 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-trusted-ca-bundle\") pod \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " Jul 02 19:11:15.792727 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.792638 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh46f\" (UniqueName: \"kubernetes.io/projected/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-kube-api-access-fh46f\") pod \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " Jul 02 19:11:15.792727 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.792664 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-oauth-serving-cert\") pod \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " Jul 02 19:11:15.792727 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.792692 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-console-oauth-config\") pod \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " Jul 02 19:11:15.792935 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.792744 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-service-ca\") pod \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " Jul 02 19:11:15.792935 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.792782 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-console-config\") pod \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\" (UID: \"a3db4cec-c581-4e0d-960a-dcf5bc1e35eb\") " Jul 02 19:11:15.793025 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.792965 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a3db4cec-c581-4e0d-960a-dcf5bc1e35eb" (UID: "a3db4cec-c581-4e0d-960a-dcf5bc1e35eb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:11:15.793277 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.793238 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-service-ca" (OuterVolumeSpecName: "service-ca") pod "a3db4cec-c581-4e0d-960a-dcf5bc1e35eb" (UID: "a3db4cec-c581-4e0d-960a-dcf5bc1e35eb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:11:15.793277 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.793244 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-console-config" (OuterVolumeSpecName: "console-config") pod "a3db4cec-c581-4e0d-960a-dcf5bc1e35eb" (UID: "a3db4cec-c581-4e0d-960a-dcf5bc1e35eb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:11:15.793428 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.793293 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a3db4cec-c581-4e0d-960a-dcf5bc1e35eb" (UID: "a3db4cec-c581-4e0d-960a-dcf5bc1e35eb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:11:15.794876 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.794854 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a3db4cec-c581-4e0d-960a-dcf5bc1e35eb" (UID: "a3db4cec-c581-4e0d-960a-dcf5bc1e35eb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:11:15.795481 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.795453 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a3db4cec-c581-4e0d-960a-dcf5bc1e35eb" (UID: "a3db4cec-c581-4e0d-960a-dcf5bc1e35eb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:11:15.795567 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.795478 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-kube-api-access-fh46f" (OuterVolumeSpecName: "kube-api-access-fh46f") pod "a3db4cec-c581-4e0d-960a-dcf5bc1e35eb" (UID: "a3db4cec-c581-4e0d-960a-dcf5bc1e35eb"). InnerVolumeSpecName "kube-api-access-fh46f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:11:15.894136 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.894107 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-trusted-ca-bundle\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:11:15.894136 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.894131 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh46f\" (UniqueName: \"kubernetes.io/projected/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-kube-api-access-fh46f\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:11:15.894136 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.894140 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-oauth-serving-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:11:15.894350 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.894148 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-console-oauth-config\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:11:15.894350 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.894157 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-service-ca\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:11:15.894350 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.894165 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-console-config\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:11:15.894350 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:15.894172 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb-console-serving-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:11:16.039105 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:16.039083 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-748c864687-drtmf"] Jul 02 19:11:16.040273 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:16.040253 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-748c864687-drtmf"] Jul 02 19:11:16.778458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:16.778343 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3db4cec-c581-4e0d-960a-dcf5bc1e35eb" path="/var/lib/kubelet/pods/a3db4cec-c581-4e0d-960a-dcf5bc1e35eb/volumes" Jul 02 19:11:42.008850 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:42.008818 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:11:42.109773 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:11:42.109740 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:08.782502 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:08.782468 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:09.551458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:09.551427 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:12.393228 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:12.393196 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:12.496405 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:12.496377 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:15.004981 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:15.004947 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:15.109476 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:15.109439 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:15.215502 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:15.215461 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:15.317751 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:15.317721 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:15.979995 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:15.979964 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5rhs6"] Jul 02 19:12:15.980301 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:15.980289 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3db4cec-c581-4e0d-960a-dcf5bc1e35eb" containerName="console" Jul 02 19:12:15.980301 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:15.980302 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3db4cec-c581-4e0d-960a-dcf5bc1e35eb" containerName="console" Jul 02 19:12:15.980390 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:15.980356 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3db4cec-c581-4e0d-960a-dcf5bc1e35eb" containerName="console" Jul 02 19:12:15.982858 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:15.982842 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5rhs6" Jul 02 19:12:15.985846 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:15.985827 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-557fb\"" Jul 02 19:12:15.985924 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:15.985843 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 02 19:12:15.986693 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:15.986673 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 02 19:12:15.992687 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:15.992668 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5rhs6"] Jul 02 19:12:16.081416 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:16.081388 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wbfq\" (UniqueName: \"kubernetes.io/projected/9ce34973-d1a7-4655-a0ea-d334ad139648-kube-api-access-8wbfq\") pod \"cert-manager-webhook-6486ffd997-5rhs6\" (UID: \"9ce34973-d1a7-4655-a0ea-d334ad139648\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5rhs6" Jul 02 19:12:16.081750 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:16.081441 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ce34973-d1a7-4655-a0ea-d334ad139648-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5rhs6\" (UID: \"9ce34973-d1a7-4655-a0ea-d334ad139648\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5rhs6" Jul 02 19:12:16.182698 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:16.182662 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wbfq\" (UniqueName: \"kubernetes.io/projected/9ce34973-d1a7-4655-a0ea-d334ad139648-kube-api-access-8wbfq\") pod \"cert-manager-webhook-6486ffd997-5rhs6\" (UID: \"9ce34973-d1a7-4655-a0ea-d334ad139648\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5rhs6" Jul 02 19:12:16.182795 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:16.182734 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ce34973-d1a7-4655-a0ea-d334ad139648-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5rhs6\" (UID: \"9ce34973-d1a7-4655-a0ea-d334ad139648\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5rhs6" Jul 02 19:12:16.191310 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:16.191281 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ce34973-d1a7-4655-a0ea-d334ad139648-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5rhs6\" (UID: \"9ce34973-d1a7-4655-a0ea-d334ad139648\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5rhs6" Jul 02 19:12:16.191914 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:16.191894 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wbfq\" (UniqueName: \"kubernetes.io/projected/9ce34973-d1a7-4655-a0ea-d334ad139648-kube-api-access-8wbfq\") pod \"cert-manager-webhook-6486ffd997-5rhs6\" (UID: \"9ce34973-d1a7-4655-a0ea-d334ad139648\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5rhs6" Jul 02 19:12:16.302439 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:16.302414 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5rhs6" Jul 02 19:12:16.433933 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:16.433910 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5rhs6"] Jul 02 19:12:16.436088 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:12:16.436055 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ce34973_d1a7_4655_a0ea_d334ad139648.slice/crio-bfc5f62c8a52e0de54e8288a34acea2a84a1fef64fb4da507a4a1dfe7a37ca72 WatchSource:0}: Error finding container bfc5f62c8a52e0de54e8288a34acea2a84a1fef64fb4da507a4a1dfe7a37ca72: Status 404 returned error can't find the container with id bfc5f62c8a52e0de54e8288a34acea2a84a1fef64fb4da507a4a1dfe7a37ca72 Jul 02 19:12:16.961336 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:16.961295 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5rhs6" event={"ID":"9ce34973-d1a7-4655-a0ea-d334ad139648","Type":"ContainerStarted","Data":"bfc5f62c8a52e0de54e8288a34acea2a84a1fef64fb4da507a4a1dfe7a37ca72"} Jul 02 19:12:17.348094 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:17.348036 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:17.606771 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:17.606681 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:17.741613 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:17.741567 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wsmfq"] Jul 02 19:12:17.747337 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:17.747193 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wsmfq" Jul 02 19:12:17.749734 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:17.749702 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-q2wm4\"" Jul 02 19:12:17.754684 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:17.754648 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wsmfq"] Jul 02 19:12:17.898236 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:17.898157 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn9p7\" (UniqueName: \"kubernetes.io/projected/2206df13-5ea4-4ed5-97a6-f9af9ce8fd62-kube-api-access-xn9p7\") pod \"cert-manager-cainjector-559cdd8c87-wsmfq\" (UID: \"2206df13-5ea4-4ed5-97a6-f9af9ce8fd62\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wsmfq" Jul 02 19:12:17.898236 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:17.898218 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2206df13-5ea4-4ed5-97a6-f9af9ce8fd62-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wsmfq\" (UID: \"2206df13-5ea4-4ed5-97a6-f9af9ce8fd62\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wsmfq" Jul 02 19:12:17.999964 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:17.999777 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn9p7\" (UniqueName: \"kubernetes.io/projected/2206df13-5ea4-4ed5-97a6-f9af9ce8fd62-kube-api-access-xn9p7\") pod \"cert-manager-cainjector-559cdd8c87-wsmfq\" (UID: \"2206df13-5ea4-4ed5-97a6-f9af9ce8fd62\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wsmfq" Jul 02 19:12:17.999964 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:17.999838 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2206df13-5ea4-4ed5-97a6-f9af9ce8fd62-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wsmfq\" (UID: \"2206df13-5ea4-4ed5-97a6-f9af9ce8fd62\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wsmfq" Jul 02 19:12:18.011007 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:18.010975 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2206df13-5ea4-4ed5-97a6-f9af9ce8fd62-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wsmfq\" (UID: \"2206df13-5ea4-4ed5-97a6-f9af9ce8fd62\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wsmfq" Jul 02 19:12:18.011572 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:18.011526 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn9p7\" (UniqueName: \"kubernetes.io/projected/2206df13-5ea4-4ed5-97a6-f9af9ce8fd62-kube-api-access-xn9p7\") pod \"cert-manager-cainjector-559cdd8c87-wsmfq\" (UID: \"2206df13-5ea4-4ed5-97a6-f9af9ce8fd62\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wsmfq" Jul 02 19:12:18.060729 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:18.060697 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wsmfq" Jul 02 19:12:18.977038 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:18.976994 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:19.044138 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:19.044113 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:19.393483 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:19.393452 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wsmfq"] Jul 02 19:12:19.397719 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:12:19.397688 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2206df13_5ea4_4ed5_97a6_f9af9ce8fd62.slice/crio-4a2aa6aeb92559422f6eaf47bd67b893f737df38e93e54fa17bd0cc739954f80 WatchSource:0}: Error finding container 4a2aa6aeb92559422f6eaf47bd67b893f737df38e93e54fa17bd0cc739954f80: Status 404 returned error can't find the container with id 4a2aa6aeb92559422f6eaf47bd67b893f737df38e93e54fa17bd0cc739954f80 Jul 02 19:12:19.977661 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:19.977625 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5rhs6" event={"ID":"9ce34973-d1a7-4655-a0ea-d334ad139648","Type":"ContainerStarted","Data":"1a35989e1aded2dd4b735dd7893bfe5682b71a571eefebae287713e56e2cc82d"} Jul 02 19:12:19.978083 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:19.977723 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5rhs6" Jul 02 19:12:19.979297 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:19.979271 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wsmfq" event={"ID":"2206df13-5ea4-4ed5-97a6-f9af9ce8fd62","Type":"ContainerStarted","Data":"0120824257b5d30571f910f45e82231b6182dc178dc203b9a1ac94d692f5f480"} Jul 02 19:12:19.979297 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:19.979301 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wsmfq" event={"ID":"2206df13-5ea4-4ed5-97a6-f9af9ce8fd62","Type":"ContainerStarted","Data":"4a2aa6aeb92559422f6eaf47bd67b893f737df38e93e54fa17bd0cc739954f80"} Jul 02 19:12:20.022771 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:20.022725 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-5rhs6" podStartSLOduration=2.150769551 podStartE2EDuration="5.022712924s" totalImagesPullingTime="2.871943373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:12:19.999795315 +0000 UTC m=+247.776812884" watchObservedRunningTime="2026-07-02 19:12:20.022712924 +0000 UTC m=+247.799730545" Jul 02 19:12:20.023570 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:20.023544 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wsmfq" podStartSLOduration=3.023534923 podStartE2EDuration="3.023534923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:12:20.021841495 +0000 UTC m=+247.798859063" watchObservedRunningTime="2026-07-02 19:12:20.023534923 +0000 UTC m=+247.800552490" Jul 02 19:12:25.605223 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:25.605180 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:25.986336 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:25.986240 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5rhs6" Jul 02 19:12:37.446855 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:37.446774 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:37.518977 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:37.518931 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:42.124532 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:42.124495 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:43.549432 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:43.549394 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:48.538013 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:48.537978 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:48.637361 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:48.637333 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:50.246538 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:50.246500 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:52.677656 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:12:52.677623 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:02.481195 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:02.481164 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:02.588223 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:02.588165 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:03.101365 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.101330 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:03.279233 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.279197 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:03.379740 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.379672 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:03.473085 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.473044 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf"] Jul 02 19:13:03.475974 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.475958 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" Jul 02 19:13:03.480273 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.480250 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 02 19:13:03.480514 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.480498 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 02 19:13:03.480651 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.480532 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 02 19:13:03.480781 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.480769 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zwcgm\"" Jul 02 19:13:03.481045 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.481028 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 02 19:13:03.498211 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.498182 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf"] Jul 02 19:13:03.578682 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.578657 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64fb8e77-6fa0-4bce-a802-d8375e44a592-tmp\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-scdsf\" (UID: \"64fb8e77-6fa0-4bce-a802-d8375e44a592\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" Jul 02 19:13:03.578824 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.578714 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f278h\" (UniqueName: \"kubernetes.io/projected/64fb8e77-6fa0-4bce-a802-d8375e44a592-kube-api-access-f278h\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-scdsf\" (UID: \"64fb8e77-6fa0-4bce-a802-d8375e44a592\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" Jul 02 19:13:03.578824 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.578794 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64fb8e77-6fa0-4bce-a802-d8375e44a592-manifests\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-scdsf\" (UID: \"64fb8e77-6fa0-4bce-a802-d8375e44a592\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" Jul 02 19:13:03.578824 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.578817 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64fb8e77-6fa0-4bce-a802-d8375e44a592-webhook-cert\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-scdsf\" (UID: \"64fb8e77-6fa0-4bce-a802-d8375e44a592\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" Jul 02 19:13:03.578941 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.578857 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64fb8e77-6fa0-4bce-a802-d8375e44a592-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-scdsf\" (UID: \"64fb8e77-6fa0-4bce-a802-d8375e44a592\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" Jul 02 19:13:03.635040 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.634974 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:03.679240 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.679217 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64fb8e77-6fa0-4bce-a802-d8375e44a592-tmp\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-scdsf\" (UID: \"64fb8e77-6fa0-4bce-a802-d8375e44a592\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" Jul 02 19:13:03.679333 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.679273 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f278h\" (UniqueName: \"kubernetes.io/projected/64fb8e77-6fa0-4bce-a802-d8375e44a592-kube-api-access-f278h\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-scdsf\" (UID: \"64fb8e77-6fa0-4bce-a802-d8375e44a592\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" Jul 02 19:13:03.679333 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.679309 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64fb8e77-6fa0-4bce-a802-d8375e44a592-manifests\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-scdsf\" (UID: \"64fb8e77-6fa0-4bce-a802-d8375e44a592\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" Jul 02 19:13:03.679422 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.679331 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64fb8e77-6fa0-4bce-a802-d8375e44a592-webhook-cert\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-scdsf\" (UID: \"64fb8e77-6fa0-4bce-a802-d8375e44a592\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" Jul 02 19:13:03.679422 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.679381 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64fb8e77-6fa0-4bce-a802-d8375e44a592-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-scdsf\" (UID: \"64fb8e77-6fa0-4bce-a802-d8375e44a592\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" Jul 02 19:13:03.679717 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.679694 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64fb8e77-6fa0-4bce-a802-d8375e44a592-manifests\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-scdsf\" (UID: \"64fb8e77-6fa0-4bce-a802-d8375e44a592\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" Jul 02 19:13:03.681789 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.681769 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64fb8e77-6fa0-4bce-a802-d8375e44a592-tmp\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-scdsf\" (UID: \"64fb8e77-6fa0-4bce-a802-d8375e44a592\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" Jul 02 19:13:03.681898 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.681876 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64fb8e77-6fa0-4bce-a802-d8375e44a592-webhook-cert\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-scdsf\" (UID: \"64fb8e77-6fa0-4bce-a802-d8375e44a592\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" Jul 02 19:13:03.681983 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.681965 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64fb8e77-6fa0-4bce-a802-d8375e44a592-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-scdsf\" (UID: \"64fb8e77-6fa0-4bce-a802-d8375e44a592\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" Jul 02 19:13:03.690984 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.690958 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f278h\" (UniqueName: \"kubernetes.io/projected/64fb8e77-6fa0-4bce-a802-d8375e44a592-kube-api-access-f278h\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-scdsf\" (UID: \"64fb8e77-6fa0-4bce-a802-d8375e44a592\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" Jul 02 19:13:03.735379 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.735350 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:03.788030 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.788008 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" Jul 02 19:13:03.833269 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.833235 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:03.922556 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.922533 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf"] Jul 02 19:13:03.925135 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:13:03.925107 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64fb8e77_6fa0_4bce_a802_d8375e44a592.slice/crio-591ba9a62152cdb3f245773297efdf51fa3b49f6fdddce186a7087b23a0adfbc WatchSource:0}: Error finding container 591ba9a62152cdb3f245773297efdf51fa3b49f6fdddce186a7087b23a0adfbc: Status 404 returned error can't find the container with id 591ba9a62152cdb3f245773297efdf51fa3b49f6fdddce186a7087b23a0adfbc Jul 02 19:13:03.925927 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.925908 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:03.990025 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:03.990003 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:04.089085 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:04.089060 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:04.155357 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:04.155327 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" event={"ID":"64fb8e77-6fa0-4bce-a802-d8375e44a592","Type":"ContainerStarted","Data":"591ba9a62152cdb3f245773297efdf51fa3b49f6fdddce186a7087b23a0adfbc"} Jul 02 19:13:06.486586 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:06.486540 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:06.588792 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:06.588749 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:07.173313 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:07.173236 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" event={"ID":"64fb8e77-6fa0-4bce-a802-d8375e44a592","Type":"ContainerStarted","Data":"d657d8cc1fb6bd07f9fd6f3b6f8f5c1220fa2b907841267cc7cb1fca9ef0c48b"} Jul 02 19:13:08.178745 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:08.178720 2562 generic.go:356] "Generic (PLEG): container finished" podID="64fb8e77-6fa0-4bce-a802-d8375e44a592" containerID="d657d8cc1fb6bd07f9fd6f3b6f8f5c1220fa2b907841267cc7cb1fca9ef0c48b" exitCode=0 Jul 02 19:13:08.179088 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:08.178797 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" event={"ID":"64fb8e77-6fa0-4bce-a802-d8375e44a592","Type":"ContainerDied","Data":"d657d8cc1fb6bd07f9fd6f3b6f8f5c1220fa2b907841267cc7cb1fca9ef0c48b"} Jul 02 19:13:09.184666 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:09.184634 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" event={"ID":"64fb8e77-6fa0-4bce-a802-d8375e44a592","Type":"ContainerStarted","Data":"dc48cadae2dbb07ed8ec6c7000632c0d46f9e4feb7260f16c4bd6fc4b79ab495"} Jul 02 19:13:09.185034 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:09.184772 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" Jul 02 19:13:09.206510 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:09.206462 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" podStartSLOduration=1.697399639 podStartE2EDuration="6.206445015s" totalImagesPullingTime="3.37912753s" totalInitContainerRuntime="1.129917846s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:13:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:13:09.204760547 +0000 UTC m=+296.981778115" watchObservedRunningTime="2026-07-02 19:13:09.206445015 +0000 UTC m=+296.983462582" Jul 02 19:13:12.620231 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:12.620206 2562 kubelet.go:1648] "Image garbage collection succeeded" Jul 02 19:13:20.191636 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:20.191584 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-scdsf" Jul 02 19:13:22.633069 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:22.633030 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:22.841827 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:22.841798 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:23.459007 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:23.458979 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:23.639343 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:23.639309 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:23.732278 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:23.732196 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:24.223526 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:24.223481 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:24.574794 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:24.574762 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:25.991494 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:25.991466 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:26.252001 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:26.251889 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:27.176677 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:27.176643 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:27.281002 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:27.280966 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:28.182310 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:28.182272 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:29.638085 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.638050 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d75bb4b7f-2vhgv"] Jul 02 19:13:29.641776 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.641752 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-2vhgv" Jul 02 19:13:29.645373 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.645336 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 02 19:13:29.645495 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.645426 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-g54kq\"" Jul 02 19:13:29.645495 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.645437 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 02 19:13:29.645495 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.645437 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:13:29.645495 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.645487 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 02 19:13:29.645694 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.645551 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 02 19:13:29.651028 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.651008 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d75bb4b7f-2vhgv"] Jul 02 19:13:29.711528 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.711505 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/392123d7-fc1b-45ab-a795-197713c10261-cert\") pod \"lws-controller-manager-7d75bb4b7f-2vhgv\" (UID: \"392123d7-fc1b-45ab-a795-197713c10261\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-2vhgv" Jul 02 19:13:29.711666 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.711534 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjjgd\" (UniqueName: \"kubernetes.io/projected/392123d7-fc1b-45ab-a795-197713c10261-kube-api-access-hjjgd\") pod \"lws-controller-manager-7d75bb4b7f-2vhgv\" (UID: \"392123d7-fc1b-45ab-a795-197713c10261\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-2vhgv" Jul 02 19:13:29.711666 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.711553 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/392123d7-fc1b-45ab-a795-197713c10261-manager-config\") pod \"lws-controller-manager-7d75bb4b7f-2vhgv\" (UID: \"392123d7-fc1b-45ab-a795-197713c10261\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-2vhgv" Jul 02 19:13:29.711666 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.711650 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/392123d7-fc1b-45ab-a795-197713c10261-metrics-cert\") pod \"lws-controller-manager-7d75bb4b7f-2vhgv\" (UID: \"392123d7-fc1b-45ab-a795-197713c10261\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-2vhgv" Jul 02 19:13:29.812092 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.812061 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/392123d7-fc1b-45ab-a795-197713c10261-metrics-cert\") pod \"lws-controller-manager-7d75bb4b7f-2vhgv\" (UID: \"392123d7-fc1b-45ab-a795-197713c10261\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-2vhgv" Jul 02 19:13:29.812207 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.812148 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/392123d7-fc1b-45ab-a795-197713c10261-cert\") pod \"lws-controller-manager-7d75bb4b7f-2vhgv\" (UID: \"392123d7-fc1b-45ab-a795-197713c10261\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-2vhgv" Jul 02 19:13:29.812207 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.812168 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjjgd\" (UniqueName: \"kubernetes.io/projected/392123d7-fc1b-45ab-a795-197713c10261-kube-api-access-hjjgd\") pod \"lws-controller-manager-7d75bb4b7f-2vhgv\" (UID: \"392123d7-fc1b-45ab-a795-197713c10261\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-2vhgv" Jul 02 19:13:29.812207 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.812185 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/392123d7-fc1b-45ab-a795-197713c10261-manager-config\") pod \"lws-controller-manager-7d75bb4b7f-2vhgv\" (UID: \"392123d7-fc1b-45ab-a795-197713c10261\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-2vhgv" Jul 02 19:13:29.812820 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.812800 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/392123d7-fc1b-45ab-a795-197713c10261-manager-config\") pod \"lws-controller-manager-7d75bb4b7f-2vhgv\" (UID: \"392123d7-fc1b-45ab-a795-197713c10261\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-2vhgv" Jul 02 19:13:29.814882 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.814857 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/392123d7-fc1b-45ab-a795-197713c10261-cert\") pod \"lws-controller-manager-7d75bb4b7f-2vhgv\" (UID: \"392123d7-fc1b-45ab-a795-197713c10261\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-2vhgv" Jul 02 19:13:29.814968 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.814857 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/392123d7-fc1b-45ab-a795-197713c10261-metrics-cert\") pod \"lws-controller-manager-7d75bb4b7f-2vhgv\" (UID: \"392123d7-fc1b-45ab-a795-197713c10261\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-2vhgv" Jul 02 19:13:29.824723 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.824701 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjjgd\" (UniqueName: \"kubernetes.io/projected/392123d7-fc1b-45ab-a795-197713c10261-kube-api-access-hjjgd\") pod \"lws-controller-manager-7d75bb4b7f-2vhgv\" (UID: \"392123d7-fc1b-45ab-a795-197713c10261\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-2vhgv" Jul 02 19:13:29.830107 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.830085 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:29.929438 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.929375 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:29.951319 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:29.951298 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-2vhgv" Jul 02 19:13:30.104090 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.104066 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d75bb4b7f-2vhgv"] Jul 02 19:13:30.105847 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:13:30.105817 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392123d7_fc1b_45ab_a795_197713c10261.slice/crio-4cbbdc14aadceca5d9830aa20a016c549203b1614132313c66d4c394443a1b3d WatchSource:0}: Error finding container 4cbbdc14aadceca5d9830aa20a016c549203b1614132313c66d4c394443a1b3d: Status 404 returned error can't find the container with id 4cbbdc14aadceca5d9830aa20a016c549203b1614132313c66d4c394443a1b3d Jul 02 19:13:30.107778 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.107760 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:13:30.282043 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.282012 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-2vhgv" event={"ID":"392123d7-fc1b-45ab-a795-197713c10261","Type":"ContainerStarted","Data":"4cbbdc14aadceca5d9830aa20a016c549203b1614132313c66d4c394443a1b3d"} Jul 02 19:13:30.736376 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.736304 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz"] Jul 02 19:13:30.739732 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.739715 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:30.742418 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.742393 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 02 19:13:30.742551 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.742481 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 02 19:13:30.742551 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.742503 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 02 19:13:30.742551 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.742514 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 02 19:13:30.742551 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.742529 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-j7xml\"" Jul 02 19:13:30.742843 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.742529 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 02 19:13:30.742906 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.742856 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 02 19:13:30.753150 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.753127 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz"] Jul 02 19:13:30.822735 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.822702 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvm6w\" (UniqueName: \"kubernetes.io/projected/97b0bd4d-3d86-4b01-b9ac-ad625173fac7-kube-api-access-nvm6w\") pod \"istiod-openshift-gateway-94bb8fbfd-hwrrz\" (UID: \"97b0bd4d-3d86-4b01-b9ac-ad625173fac7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:30.822735 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.822738 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97b0bd4d-3d86-4b01-b9ac-ad625173fac7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-hwrrz\" (UID: \"97b0bd4d-3d86-4b01-b9ac-ad625173fac7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:30.822915 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.822763 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97b0bd4d-3d86-4b01-b9ac-ad625173fac7-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-hwrrz\" (UID: \"97b0bd4d-3d86-4b01-b9ac-ad625173fac7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:30.822915 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.822822 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97b0bd4d-3d86-4b01-b9ac-ad625173fac7-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-hwrrz\" (UID: \"97b0bd4d-3d86-4b01-b9ac-ad625173fac7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:30.822915 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.822838 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97b0bd4d-3d86-4b01-b9ac-ad625173fac7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-hwrrz\" (UID: \"97b0bd4d-3d86-4b01-b9ac-ad625173fac7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:30.822915 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.822854 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97b0bd4d-3d86-4b01-b9ac-ad625173fac7-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-hwrrz\" (UID: \"97b0bd4d-3d86-4b01-b9ac-ad625173fac7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:30.822915 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.822870 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97b0bd4d-3d86-4b01-b9ac-ad625173fac7-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-hwrrz\" (UID: \"97b0bd4d-3d86-4b01-b9ac-ad625173fac7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:30.923655 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.923624 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvm6w\" (UniqueName: \"kubernetes.io/projected/97b0bd4d-3d86-4b01-b9ac-ad625173fac7-kube-api-access-nvm6w\") pod \"istiod-openshift-gateway-94bb8fbfd-hwrrz\" (UID: \"97b0bd4d-3d86-4b01-b9ac-ad625173fac7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:30.923798 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.923664 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97b0bd4d-3d86-4b01-b9ac-ad625173fac7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-hwrrz\" (UID: \"97b0bd4d-3d86-4b01-b9ac-ad625173fac7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:30.923798 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.923690 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97b0bd4d-3d86-4b01-b9ac-ad625173fac7-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-hwrrz\" (UID: \"97b0bd4d-3d86-4b01-b9ac-ad625173fac7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:30.923798 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.923740 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97b0bd4d-3d86-4b01-b9ac-ad625173fac7-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-hwrrz\" (UID: \"97b0bd4d-3d86-4b01-b9ac-ad625173fac7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:30.923798 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.923763 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97b0bd4d-3d86-4b01-b9ac-ad625173fac7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-hwrrz\" (UID: \"97b0bd4d-3d86-4b01-b9ac-ad625173fac7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:30.923798 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.923792 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97b0bd4d-3d86-4b01-b9ac-ad625173fac7-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-hwrrz\" (UID: \"97b0bd4d-3d86-4b01-b9ac-ad625173fac7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:30.924105 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.923816 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97b0bd4d-3d86-4b01-b9ac-ad625173fac7-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-hwrrz\" (UID: \"97b0bd4d-3d86-4b01-b9ac-ad625173fac7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:30.924295 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.924261 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/97b0bd4d-3d86-4b01-b9ac-ad625173fac7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-hwrrz\" (UID: \"97b0bd4d-3d86-4b01-b9ac-ad625173fac7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:30.926238 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.926202 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/97b0bd4d-3d86-4b01-b9ac-ad625173fac7-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-hwrrz\" (UID: \"97b0bd4d-3d86-4b01-b9ac-ad625173fac7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:30.926796 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.926776 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/97b0bd4d-3d86-4b01-b9ac-ad625173fac7-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-hwrrz\" (UID: \"97b0bd4d-3d86-4b01-b9ac-ad625173fac7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:30.926898 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.926826 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/97b0bd4d-3d86-4b01-b9ac-ad625173fac7-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-hwrrz\" (UID: \"97b0bd4d-3d86-4b01-b9ac-ad625173fac7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:30.926898 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.926890 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/97b0bd4d-3d86-4b01-b9ac-ad625173fac7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-hwrrz\" (UID: \"97b0bd4d-3d86-4b01-b9ac-ad625173fac7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:30.931663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.931638 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97b0bd4d-3d86-4b01-b9ac-ad625173fac7-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-hwrrz\" (UID: \"97b0bd4d-3d86-4b01-b9ac-ad625173fac7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:30.931832 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:30.931814 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvm6w\" (UniqueName: \"kubernetes.io/projected/97b0bd4d-3d86-4b01-b9ac-ad625173fac7-kube-api-access-nvm6w\") pod \"istiod-openshift-gateway-94bb8fbfd-hwrrz\" (UID: \"97b0bd4d-3d86-4b01-b9ac-ad625173fac7\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:31.049374 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:31.049348 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:31.050310 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:31.050289 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:31.150973 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:31.150942 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:31.181388 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:31.181368 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz"] Jul 02 19:13:31.183530 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:13:31.183500 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b0bd4d_3d86_4b01_b9ac_ad625173fac7.slice/crio-4d8aeef02caf2821e55cdbb5d1df35cd3ac21480ebe187619ec75f865696a4c0 WatchSource:0}: Error finding container 4d8aeef02caf2821e55cdbb5d1df35cd3ac21480ebe187619ec75f865696a4c0: Status 404 returned error can't find the container with id 4d8aeef02caf2821e55cdbb5d1df35cd3ac21480ebe187619ec75f865696a4c0 Jul 02 19:13:31.288202 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:31.288158 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" event={"ID":"97b0bd4d-3d86-4b01-b9ac-ad625173fac7","Type":"ContainerStarted","Data":"4d8aeef02caf2821e55cdbb5d1df35cd3ac21480ebe187619ec75f865696a4c0"} Jul 02 19:13:32.195977 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:32.195943 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:32.533434 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:32.533400 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:32.637825 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:32.637795 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:33.832629 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:33.832574 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:33.951235 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:33.951135 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:34.122855 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:34.122775 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:34.268447 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:34.268411 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:35.163087 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:35.163040 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:35.322086 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:35.322046 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:37.335359 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:37.335317 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 02 19:13:37.335680 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:37.335408 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 02 19:13:38.323501 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:38.323461 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" event={"ID":"97b0bd4d-3d86-4b01-b9ac-ad625173fac7","Type":"ContainerStarted","Data":"65b67b9c82156a818804624f36b39fa40fae11b805e4ec91202e2dcc9eae23b1"} Jul 02 19:13:38.323723 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:38.323674 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:38.325272 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:38.325251 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" Jul 02 19:13:38.344269 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:13:38.344220 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-hwrrz" podStartSLOduration=2.194933519 podStartE2EDuration="8.34420461s" totalImagesPullingTime="6.149271091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:13:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:13:38.342761165 +0000 UTC m=+326.119778745" watchObservedRunningTime="2026-07-02 19:13:38.34420461 +0000 UTC m=+326.121222178" Jul 02 19:14:08.462671 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:14:08.462630 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-2vhgv" event={"ID":"392123d7-fc1b-45ab-a795-197713c10261","Type":"ContainerStarted","Data":"a84c998ef03b5026f6008c3296a9beec7cfc628c4bc17d639f35eded9dfcba3f"} Jul 02 19:14:08.463074 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:14:08.462749 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-2vhgv" Jul 02 19:14:08.481130 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:14:08.481087 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-2vhgv" podStartSLOduration=2.120790828 podStartE2EDuration="39.481071276s" totalImagesPullingTime="37.360280448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:13:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:14:08.480313778 +0000 UTC m=+356.257331368" watchObservedRunningTime="2026-07-02 19:14:08.481071276 +0000 UTC m=+356.258088842" Jul 02 19:14:19.470142 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:14:19.470108 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-2vhgv" Jul 02 19:14:24.184054 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:14:24.184010 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:14:24.301094 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:14:24.301063 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:14:24.431582 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:14:24.431549 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:14:25.123138 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:14:25.123106 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:14:25.329793 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:14:25.329757 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:06.964886 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:06.964850 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:11.179118 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:11.179079 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:13.554116 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:13.554078 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:14.200753 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:14.200712 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:14.464849 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:14.464771 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:14.549684 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:14.549654 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:16.202140 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:16.202095 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:17.600368 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:17.600336 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:17.764602 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:17.764569 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:19.414183 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:19.414148 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:22.581437 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:22.581401 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:22.683190 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:22.683147 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:26.372121 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:26.372091 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:26.472381 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:26.472350 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:27.355574 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:27.355538 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:27.460456 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:27.460424 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:29.814878 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:29.814849 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:30.516230 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:30.516201 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:32.169796 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:32.169759 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:32.230230 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:32.230201 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:32.362393 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:32.362356 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:32.466102 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:32.466027 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:35.522364 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:35.522329 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:35.623397 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:35.623367 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:40.019700 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.019663 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4j9xm"] Jul 02 19:15:40.022285 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.022267 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4j9xm" Jul 02 19:15:40.025208 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.025175 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-m9ktd\"" Jul 02 19:15:40.025320 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.025305 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 02 19:15:40.025686 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.025668 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 02 19:15:40.038822 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.038795 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4j9xm"] Jul 02 19:15:40.130792 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.130764 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4j9xm"] Jul 02 19:15:40.131077 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:15:40.131055 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-crkxj], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4j9xm" podUID="9dc7636b-1126-4000-b09a-6d5eeda296c1" Jul 02 19:15:40.136105 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.136079 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9dc7636b-1126-4000-b09a-6d5eeda296c1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4j9xm\" (UID: \"9dc7636b-1126-4000-b09a-6d5eeda296c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4j9xm" Jul 02 19:15:40.136216 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.136166 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crkxj\" (UniqueName: \"kubernetes.io/projected/9dc7636b-1126-4000-b09a-6d5eeda296c1-kube-api-access-crkxj\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4j9xm\" (UID: \"9dc7636b-1126-4000-b09a-6d5eeda296c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4j9xm" Jul 02 19:15:40.237117 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.237087 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crkxj\" (UniqueName: \"kubernetes.io/projected/9dc7636b-1126-4000-b09a-6d5eeda296c1-kube-api-access-crkxj\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4j9xm\" (UID: \"9dc7636b-1126-4000-b09a-6d5eeda296c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4j9xm" Jul 02 19:15:40.237264 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.237183 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9dc7636b-1126-4000-b09a-6d5eeda296c1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4j9xm\" (UID: \"9dc7636b-1126-4000-b09a-6d5eeda296c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4j9xm" Jul 02 19:15:40.237560 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.237543 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9dc7636b-1126-4000-b09a-6d5eeda296c1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4j9xm\" (UID: \"9dc7636b-1126-4000-b09a-6d5eeda296c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4j9xm" Jul 02 19:15:40.250400 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.250374 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crkxj\" (UniqueName: \"kubernetes.io/projected/9dc7636b-1126-4000-b09a-6d5eeda296c1-kube-api-access-crkxj\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-4j9xm\" (UID: \"9dc7636b-1126-4000-b09a-6d5eeda296c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4j9xm" Jul 02 19:15:40.580225 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.580187 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4j9xm"] Jul 02 19:15:40.589781 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.589753 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4j9xm"] Jul 02 19:15:40.637207 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.637166 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb"] Jul 02 19:15:40.640937 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.640902 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb" Jul 02 19:15:40.675123 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.675098 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb"] Jul 02 19:15:40.741923 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.741889 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7eed460f-d1ed-496c-9fce-5d8d53933f1d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ljgfb\" (UID: \"7eed460f-d1ed-496c-9fce-5d8d53933f1d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb" Jul 02 19:15:40.741923 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.741924 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s5fl\" (UniqueName: \"kubernetes.io/projected/7eed460f-d1ed-496c-9fce-5d8d53933f1d-kube-api-access-8s5fl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ljgfb\" (UID: \"7eed460f-d1ed-496c-9fce-5d8d53933f1d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb" Jul 02 19:15:40.843309 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.843235 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7eed460f-d1ed-496c-9fce-5d8d53933f1d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ljgfb\" (UID: \"7eed460f-d1ed-496c-9fce-5d8d53933f1d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb" Jul 02 19:15:40.843309 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.843270 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s5fl\" (UniqueName: \"kubernetes.io/projected/7eed460f-d1ed-496c-9fce-5d8d53933f1d-kube-api-access-8s5fl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ljgfb\" (UID: \"7eed460f-d1ed-496c-9fce-5d8d53933f1d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb" Jul 02 19:15:40.843637 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.843618 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7eed460f-d1ed-496c-9fce-5d8d53933f1d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ljgfb\" (UID: \"7eed460f-d1ed-496c-9fce-5d8d53933f1d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb" Jul 02 19:15:40.856729 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.856705 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s5fl\" (UniqueName: \"kubernetes.io/projected/7eed460f-d1ed-496c-9fce-5d8d53933f1d-kube-api-access-8s5fl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ljgfb\" (UID: \"7eed460f-d1ed-496c-9fce-5d8d53933f1d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb" Jul 02 19:15:40.874212 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.874180 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4j9xm" Jul 02 19:15:40.880065 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.880049 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4j9xm" Jul 02 19:15:40.883373 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:15:40.883353 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-4j9xm\" is forbidden: User \"system:node:ip-10-0-133-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-147.ec2.internal' and this object" podUID="9dc7636b-1126-4000-b09a-6d5eeda296c1" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4j9xm" Jul 02 19:15:40.952297 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:40.952275 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb" Jul 02 19:15:41.045476 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:41.045449 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crkxj\" (UniqueName: \"kubernetes.io/projected/9dc7636b-1126-4000-b09a-6d5eeda296c1-kube-api-access-crkxj\") pod \"9dc7636b-1126-4000-b09a-6d5eeda296c1\" (UID: \"9dc7636b-1126-4000-b09a-6d5eeda296c1\") " Jul 02 19:15:41.045883 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:41.045507 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9dc7636b-1126-4000-b09a-6d5eeda296c1-extensions-socket-volume\") pod \"9dc7636b-1126-4000-b09a-6d5eeda296c1\" (UID: \"9dc7636b-1126-4000-b09a-6d5eeda296c1\") " Jul 02 19:15:41.045883 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:41.045829 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc7636b-1126-4000-b09a-6d5eeda296c1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9dc7636b-1126-4000-b09a-6d5eeda296c1" (UID: "9dc7636b-1126-4000-b09a-6d5eeda296c1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:15:41.047905 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:41.047877 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc7636b-1126-4000-b09a-6d5eeda296c1-kube-api-access-crkxj" (OuterVolumeSpecName: "kube-api-access-crkxj") pod "9dc7636b-1126-4000-b09a-6d5eeda296c1" (UID: "9dc7636b-1126-4000-b09a-6d5eeda296c1"). InnerVolumeSpecName "kube-api-access-crkxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:15:41.085972 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:41.085950 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb"] Jul 02 19:15:41.088435 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:15:41.088406 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eed460f_d1ed_496c_9fce_5d8d53933f1d.slice/crio-b0ae4f7c865dd4cf59b0e91f3eb8c831dd87714a0ce3faf9015df812644fd5b7 WatchSource:0}: Error finding container b0ae4f7c865dd4cf59b0e91f3eb8c831dd87714a0ce3faf9015df812644fd5b7: Status 404 returned error can't find the container with id b0ae4f7c865dd4cf59b0e91f3eb8c831dd87714a0ce3faf9015df812644fd5b7 Jul 02 19:15:41.146230 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:41.146203 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crkxj\" (UniqueName: \"kubernetes.io/projected/9dc7636b-1126-4000-b09a-6d5eeda296c1-kube-api-access-crkxj\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:15:41.146326 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:41.146229 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9dc7636b-1126-4000-b09a-6d5eeda296c1-extensions-socket-volume\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:15:41.797058 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:41.797029 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:41.880788 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:41.880754 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb" event={"ID":"7eed460f-d1ed-496c-9fce-5d8d53933f1d","Type":"ContainerStarted","Data":"b0ae4f7c865dd4cf59b0e91f3eb8c831dd87714a0ce3faf9015df812644fd5b7"} Jul 02 19:15:41.880951 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:41.880797 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4j9xm" Jul 02 19:15:41.883281 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:15:41.883253 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-4j9xm\" is forbidden: User \"system:node:ip-10-0-133-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-147.ec2.internal' and this object" podUID="9dc7636b-1126-4000-b09a-6d5eeda296c1" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4j9xm" Jul 02 19:15:41.890554 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:15:41.890521 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-4j9xm\" is forbidden: User \"system:node:ip-10-0-133-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-147.ec2.internal' and this object" podUID="9dc7636b-1126-4000-b09a-6d5eeda296c1" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4j9xm" Jul 02 19:15:41.898739 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:41.898715 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:42.206776 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:42.206689 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:42.334819 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:42.334775 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:42.775834 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:15:42.775796 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-4j9xm\" is forbidden: User \"system:node:ip-10-0-133-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-147.ec2.internal' and this object" podUID="9dc7636b-1126-4000-b09a-6d5eeda296c1" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-4j9xm" Jul 02 19:15:42.778606 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:42.778568 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dc7636b-1126-4000-b09a-6d5eeda296c1" path="/var/lib/kubelet/pods/9dc7636b-1126-4000-b09a-6d5eeda296c1/volumes" Jul 02 19:15:43.091842 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:43.091752 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:43.192324 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:43.192274 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:43.943927 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:43.943897 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6756cc7865-5jlzt"] Jul 02 19:15:43.946428 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:43.946411 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:43.951255 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:43.951206 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 02 19:15:43.951401 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:43.951376 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 02 19:15:43.952493 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:43.952439 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 02 19:15:43.952493 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:43.952460 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 02 19:15:43.952493 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:43.952477 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 02 19:15:43.952769 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:43.952464 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kq4f6\"" Jul 02 19:15:43.952769 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:43.952460 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 02 19:15:43.952769 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:43.952461 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 02 19:15:43.959659 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:43.959640 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 02 19:15:43.967387 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:43.967368 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6756cc7865-5jlzt"] Jul 02 19:15:44.022820 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.022792 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:44.075006 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.074969 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a244ef32-fc8f-4eb5-9866-e9d588e8fb54-console-oauth-config\") pod \"console-6756cc7865-5jlzt\" (UID: \"a244ef32-fc8f-4eb5-9866-e9d588e8fb54\") " pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:44.075006 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.074998 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a244ef32-fc8f-4eb5-9866-e9d588e8fb54-console-config\") pod \"console-6756cc7865-5jlzt\" (UID: \"a244ef32-fc8f-4eb5-9866-e9d588e8fb54\") " pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:44.075209 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.075036 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2m76\" (UniqueName: \"kubernetes.io/projected/a244ef32-fc8f-4eb5-9866-e9d588e8fb54-kube-api-access-s2m76\") pod \"console-6756cc7865-5jlzt\" (UID: \"a244ef32-fc8f-4eb5-9866-e9d588e8fb54\") " pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:44.075209 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.075088 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a244ef32-fc8f-4eb5-9866-e9d588e8fb54-service-ca\") pod \"console-6756cc7865-5jlzt\" (UID: \"a244ef32-fc8f-4eb5-9866-e9d588e8fb54\") " pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:44.075209 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.075116 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a244ef32-fc8f-4eb5-9866-e9d588e8fb54-oauth-serving-cert\") pod \"console-6756cc7865-5jlzt\" (UID: \"a244ef32-fc8f-4eb5-9866-e9d588e8fb54\") " pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:44.075209 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.075150 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a244ef32-fc8f-4eb5-9866-e9d588e8fb54-console-serving-cert\") pod \"console-6756cc7865-5jlzt\" (UID: \"a244ef32-fc8f-4eb5-9866-e9d588e8fb54\") " pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:44.075209 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.075168 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a244ef32-fc8f-4eb5-9866-e9d588e8fb54-trusted-ca-bundle\") pod \"console-6756cc7865-5jlzt\" (UID: \"a244ef32-fc8f-4eb5-9866-e9d588e8fb54\") " pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:44.126129 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.126105 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:44.176284 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.176262 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a244ef32-fc8f-4eb5-9866-e9d588e8fb54-console-oauth-config\") pod \"console-6756cc7865-5jlzt\" (UID: \"a244ef32-fc8f-4eb5-9866-e9d588e8fb54\") " pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:44.176354 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.176288 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a244ef32-fc8f-4eb5-9866-e9d588e8fb54-console-config\") pod \"console-6756cc7865-5jlzt\" (UID: \"a244ef32-fc8f-4eb5-9866-e9d588e8fb54\") " pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:44.176354 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.176308 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2m76\" (UniqueName: \"kubernetes.io/projected/a244ef32-fc8f-4eb5-9866-e9d588e8fb54-kube-api-access-s2m76\") pod \"console-6756cc7865-5jlzt\" (UID: \"a244ef32-fc8f-4eb5-9866-e9d588e8fb54\") " pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:44.176354 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.176327 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a244ef32-fc8f-4eb5-9866-e9d588e8fb54-service-ca\") pod \"console-6756cc7865-5jlzt\" (UID: \"a244ef32-fc8f-4eb5-9866-e9d588e8fb54\") " pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:44.176488 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.176400 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a244ef32-fc8f-4eb5-9866-e9d588e8fb54-oauth-serving-cert\") pod \"console-6756cc7865-5jlzt\" (UID: \"a244ef32-fc8f-4eb5-9866-e9d588e8fb54\") " pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:44.176488 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.176436 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a244ef32-fc8f-4eb5-9866-e9d588e8fb54-console-serving-cert\") pod \"console-6756cc7865-5jlzt\" (UID: \"a244ef32-fc8f-4eb5-9866-e9d588e8fb54\") " pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:44.176488 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.176461 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a244ef32-fc8f-4eb5-9866-e9d588e8fb54-trusted-ca-bundle\") pod \"console-6756cc7865-5jlzt\" (UID: \"a244ef32-fc8f-4eb5-9866-e9d588e8fb54\") " pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:44.177174 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.177095 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a244ef32-fc8f-4eb5-9866-e9d588e8fb54-service-ca\") pod \"console-6756cc7865-5jlzt\" (UID: \"a244ef32-fc8f-4eb5-9866-e9d588e8fb54\") " pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:44.177174 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.177136 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a244ef32-fc8f-4eb5-9866-e9d588e8fb54-console-config\") pod \"console-6756cc7865-5jlzt\" (UID: \"a244ef32-fc8f-4eb5-9866-e9d588e8fb54\") " pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:44.177352 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.177333 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a244ef32-fc8f-4eb5-9866-e9d588e8fb54-trusted-ca-bundle\") pod \"console-6756cc7865-5jlzt\" (UID: \"a244ef32-fc8f-4eb5-9866-e9d588e8fb54\") " pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:44.177390 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.177335 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a244ef32-fc8f-4eb5-9866-e9d588e8fb54-oauth-serving-cert\") pod \"console-6756cc7865-5jlzt\" (UID: \"a244ef32-fc8f-4eb5-9866-e9d588e8fb54\") " pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:44.179054 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.179033 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a244ef32-fc8f-4eb5-9866-e9d588e8fb54-console-oauth-config\") pod \"console-6756cc7865-5jlzt\" (UID: \"a244ef32-fc8f-4eb5-9866-e9d588e8fb54\") " pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:44.179137 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.179121 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a244ef32-fc8f-4eb5-9866-e9d588e8fb54-console-serving-cert\") pod \"console-6756cc7865-5jlzt\" (UID: \"a244ef32-fc8f-4eb5-9866-e9d588e8fb54\") " pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:44.188459 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.188435 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2m76\" (UniqueName: \"kubernetes.io/projected/a244ef32-fc8f-4eb5-9866-e9d588e8fb54-kube-api-access-s2m76\") pod \"console-6756cc7865-5jlzt\" (UID: \"a244ef32-fc8f-4eb5-9866-e9d588e8fb54\") " pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:44.256645 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.256564 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:44.408882 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.408860 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6756cc7865-5jlzt"] Jul 02 19:15:44.411004 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:15:44.410977 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda244ef32_fc8f_4eb5_9866_e9d588e8fb54.slice/crio-600fa8858994bbd9559ce91b6e093b7a5aa5ff8869b9a168e5f6972b3efd5f18 WatchSource:0}: Error finding container 600fa8858994bbd9559ce91b6e093b7a5aa5ff8869b9a168e5f6972b3efd5f18: Status 404 returned error can't find the container with id 600fa8858994bbd9559ce91b6e093b7a5aa5ff8869b9a168e5f6972b3efd5f18 Jul 02 19:15:44.896373 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.896343 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6756cc7865-5jlzt" event={"ID":"a244ef32-fc8f-4eb5-9866-e9d588e8fb54","Type":"ContainerStarted","Data":"f9c9002928c607383c64247a652886fed48b8c4141b150354f3acd2c312dd961"} Jul 02 19:15:44.896575 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.896385 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6756cc7865-5jlzt" event={"ID":"a244ef32-fc8f-4eb5-9866-e9d588e8fb54","Type":"ContainerStarted","Data":"600fa8858994bbd9559ce91b6e093b7a5aa5ff8869b9a168e5f6972b3efd5f18"} Jul 02 19:15:44.921161 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:44.921098 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6756cc7865-5jlzt" podStartSLOduration=1.92107644 podStartE2EDuration="1.92107644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:15:44.918915203 +0000 UTC m=+452.695932790" watchObservedRunningTime="2026-07-02 19:15:44.92107644 +0000 UTC m=+452.698094008" Jul 02 19:15:45.186449 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:45.186361 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:45.286486 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:45.286454 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:46.908654 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:46.908619 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb" event={"ID":"7eed460f-d1ed-496c-9fce-5d8d53933f1d","Type":"ContainerStarted","Data":"9b7430f52b41bda9992e913fea32caf039b92dc8b93e0b9f3e93c0760dd00a32"} Jul 02 19:15:46.909029 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:46.908760 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb" Jul 02 19:15:46.935290 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:46.935242 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb" podStartSLOduration=1.842366761 podStartE2EDuration="6.935226777s" totalImagesPullingTime="5.092860016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:15:46.934047994 +0000 UTC m=+454.711065561" watchObservedRunningTime="2026-07-02 19:15:46.935226777 +0000 UTC m=+454.712244334" Jul 02 19:15:49.996788 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:49.996745 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:50.097449 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:50.097413 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:50.197215 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:50.197185 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:50.261335 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:50.261262 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:54.257038 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:54.257004 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:54.257399 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:54.257058 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:54.261447 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:54.261425 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:54.947026 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:54.947001 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6756cc7865-5jlzt" Jul 02 19:15:55.130920 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:55.130890 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:55.236885 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:55.236801 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:55.332713 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:55.332681 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:57.916390 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:15:57.916361 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb" Jul 02 19:16:06.044266 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:06.044230 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb"] Jul 02 19:16:06.044863 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:06.044458 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb" podUID="7eed460f-d1ed-496c-9fce-5d8d53933f1d" containerName="manager" containerID="cri-o://9b7430f52b41bda9992e913fea32caf039b92dc8b93e0b9f3e93c0760dd00a32" gracePeriod=10 Jul 02 19:16:06.296727 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:06.296666 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb" Jul 02 19:16:06.373346 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:06.373310 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7eed460f-d1ed-496c-9fce-5d8d53933f1d-extensions-socket-volume\") pod \"7eed460f-d1ed-496c-9fce-5d8d53933f1d\" (UID: \"7eed460f-d1ed-496c-9fce-5d8d53933f1d\") " Jul 02 19:16:06.373514 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:06.373364 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s5fl\" (UniqueName: \"kubernetes.io/projected/7eed460f-d1ed-496c-9fce-5d8d53933f1d-kube-api-access-8s5fl\") pod \"7eed460f-d1ed-496c-9fce-5d8d53933f1d\" (UID: \"7eed460f-d1ed-496c-9fce-5d8d53933f1d\") " Jul 02 19:16:06.373835 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:06.373812 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eed460f-d1ed-496c-9fce-5d8d53933f1d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7eed460f-d1ed-496c-9fce-5d8d53933f1d" (UID: "7eed460f-d1ed-496c-9fce-5d8d53933f1d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:16:06.375635 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:06.375618 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eed460f-d1ed-496c-9fce-5d8d53933f1d-kube-api-access-8s5fl" (OuterVolumeSpecName: "kube-api-access-8s5fl") pod "7eed460f-d1ed-496c-9fce-5d8d53933f1d" (UID: "7eed460f-d1ed-496c-9fce-5d8d53933f1d"). InnerVolumeSpecName "kube-api-access-8s5fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:16:06.474180 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:06.474152 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s5fl\" (UniqueName: \"kubernetes.io/projected/7eed460f-d1ed-496c-9fce-5d8d53933f1d-kube-api-access-8s5fl\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:16:06.474180 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:06.474175 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7eed460f-d1ed-496c-9fce-5d8d53933f1d-extensions-socket-volume\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:16:07.001875 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:07.001847 2562 generic.go:356] "Generic (PLEG): container finished" podID="7eed460f-d1ed-496c-9fce-5d8d53933f1d" containerID="9b7430f52b41bda9992e913fea32caf039b92dc8b93e0b9f3e93c0760dd00a32" exitCode=0 Jul 02 19:16:07.002065 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:07.001898 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb" Jul 02 19:16:07.002065 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:07.001936 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb" event={"ID":"7eed460f-d1ed-496c-9fce-5d8d53933f1d","Type":"ContainerDied","Data":"9b7430f52b41bda9992e913fea32caf039b92dc8b93e0b9f3e93c0760dd00a32"} Jul 02 19:16:07.002065 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:07.001999 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb" event={"ID":"7eed460f-d1ed-496c-9fce-5d8d53933f1d","Type":"ContainerDied","Data":"b0ae4f7c865dd4cf59b0e91f3eb8c831dd87714a0ce3faf9015df812644fd5b7"} Jul 02 19:16:07.002065 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:07.002023 2562 scope.go:117] "RemoveContainer" containerID="9b7430f52b41bda9992e913fea32caf039b92dc8b93e0b9f3e93c0760dd00a32" Jul 02 19:16:07.011668 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:07.011645 2562 scope.go:117] "RemoveContainer" containerID="9b7430f52b41bda9992e913fea32caf039b92dc8b93e0b9f3e93c0760dd00a32" Jul 02 19:16:07.011937 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:16:07.011919 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b7430f52b41bda9992e913fea32caf039b92dc8b93e0b9f3e93c0760dd00a32\": container with ID starting with 9b7430f52b41bda9992e913fea32caf039b92dc8b93e0b9f3e93c0760dd00a32 not found: ID does not exist" containerID="9b7430f52b41bda9992e913fea32caf039b92dc8b93e0b9f3e93c0760dd00a32" Jul 02 19:16:07.012018 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:07.011945 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7430f52b41bda9992e913fea32caf039b92dc8b93e0b9f3e93c0760dd00a32"} err="failed to get container status \"9b7430f52b41bda9992e913fea32caf039b92dc8b93e0b9f3e93c0760dd00a32\": rpc error: code = NotFound desc = could not find container \"9b7430f52b41bda9992e913fea32caf039b92dc8b93e0b9f3e93c0760dd00a32\": container with ID starting with 9b7430f52b41bda9992e913fea32caf039b92dc8b93e0b9f3e93c0760dd00a32 not found: ID does not exist" Jul 02 19:16:07.025168 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:07.025140 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb"] Jul 02 19:16:07.028747 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:07.028726 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ljgfb"] Jul 02 19:16:08.777231 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:08.777201 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eed460f-d1ed-496c-9fce-5d8d53933f1d" path="/var/lib/kubelet/pods/7eed460f-d1ed-496c-9fce-5d8d53933f1d/volumes" Jul 02 19:16:22.446786 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:22.446746 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:16:22.545242 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:22.545211 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:16:25.495748 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:25.495717 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tktk7"] Jul 02 19:16:25.496172 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:25.496072 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eed460f-d1ed-496c-9fce-5d8d53933f1d" containerName="manager" Jul 02 19:16:25.496172 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:25.496086 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eed460f-d1ed-496c-9fce-5d8d53933f1d" containerName="manager" Jul 02 19:16:25.496307 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:25.496195 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eed460f-d1ed-496c-9fce-5d8d53933f1d" containerName="manager" Jul 02 19:16:25.499219 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:25.499190 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:16:25.500088 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:25.500072 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-tktk7" Jul 02 19:16:25.502910 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:25.502888 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 02 19:16:25.503024 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:25.502908 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 02 19:16:25.503898 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:25.503881 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 02 19:16:25.503976 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:25.503903 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pfnst\"" Jul 02 19:16:25.508710 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:25.508685 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tktk7"] Jul 02 19:16:25.595604 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:25.595560 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tktk7"] Jul 02 19:16:25.598687 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:25.598647 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:16:25.636763 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:25.636733 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm4ls\" (UniqueName: \"kubernetes.io/projected/a0053e6d-76a7-41ea-94e2-03d3ada00909-kube-api-access-gm4ls\") pod \"limitador-limitador-64c94fd64f-tktk7\" (UID: \"a0053e6d-76a7-41ea-94e2-03d3ada00909\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tktk7" Jul 02 19:16:25.636885 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:25.636850 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a0053e6d-76a7-41ea-94e2-03d3ada00909-config-file\") pod \"limitador-limitador-64c94fd64f-tktk7\" (UID: \"a0053e6d-76a7-41ea-94e2-03d3ada00909\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tktk7" Jul 02 19:16:25.700187 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:25.700148 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:16:25.737909 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:25.737876 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a0053e6d-76a7-41ea-94e2-03d3ada00909-config-file\") pod \"limitador-limitador-64c94fd64f-tktk7\" (UID: \"a0053e6d-76a7-41ea-94e2-03d3ada00909\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tktk7" Jul 02 19:16:25.738039 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:25.737960 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm4ls\" (UniqueName: \"kubernetes.io/projected/a0053e6d-76a7-41ea-94e2-03d3ada00909-kube-api-access-gm4ls\") pod \"limitador-limitador-64c94fd64f-tktk7\" (UID: \"a0053e6d-76a7-41ea-94e2-03d3ada00909\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tktk7" Jul 02 19:16:25.738446 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:25.738428 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a0053e6d-76a7-41ea-94e2-03d3ada00909-config-file\") pod \"limitador-limitador-64c94fd64f-tktk7\" (UID: \"a0053e6d-76a7-41ea-94e2-03d3ada00909\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tktk7" Jul 02 19:16:25.746657 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:25.746574 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm4ls\" (UniqueName: \"kubernetes.io/projected/a0053e6d-76a7-41ea-94e2-03d3ada00909-kube-api-access-gm4ls\") pod \"limitador-limitador-64c94fd64f-tktk7\" (UID: \"a0053e6d-76a7-41ea-94e2-03d3ada00909\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tktk7" Jul 02 19:16:25.811579 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:25.811554 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-tktk7" Jul 02 19:16:25.949370 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:25.949246 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tktk7"] Jul 02 19:16:25.951851 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:16:25.951815 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0053e6d_76a7_41ea_94e2_03d3ada00909.slice/crio-8bc86ea7e897981e087318fbf884055cab8dd55d4b90d17bdc4ca85477a329f6 WatchSource:0}: Error finding container 8bc86ea7e897981e087318fbf884055cab8dd55d4b90d17bdc4ca85477a329f6: Status 404 returned error can't find the container with id 8bc86ea7e897981e087318fbf884055cab8dd55d4b90d17bdc4ca85477a329f6 Jul 02 19:16:26.092991 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:26.092959 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-tktk7" event={"ID":"a0053e6d-76a7-41ea-94e2-03d3ada00909","Type":"ContainerStarted","Data":"8bc86ea7e897981e087318fbf884055cab8dd55d4b90d17bdc4ca85477a329f6"} Jul 02 19:16:26.348203 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:26.348114 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:16:26.409990 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:26.409962 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:16:29.109694 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:29.109660 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-tktk7" event={"ID":"a0053e6d-76a7-41ea-94e2-03d3ada00909","Type":"ContainerStarted","Data":"d755e72d3a1cb56fa82ba8cfecce7ef0a62b4fe5dc655b52b261562925c9edcf"} Jul 02 19:16:29.110054 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:29.109790 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-tktk7" Jul 02 19:16:29.129378 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:29.129334 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-tktk7" podStartSLOduration=1.5449148099999999 podStartE2EDuration="4.129319446s" totalImagesPullingTime="2.584404636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:16:29.127794003 +0000 UTC m=+496.904811575" watchObservedRunningTime="2026-07-02 19:16:29.129319446 +0000 UTC m=+496.906337014" Jul 02 19:16:37.419353 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:37.419317 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:16:37.520961 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:37.520931 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:16:40.116045 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:40.116015 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-tktk7" Jul 02 19:16:40.849340 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:40.849309 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tktk7"] Jul 02 19:16:40.849536 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:40.849491 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-tktk7" podUID="a0053e6d-76a7-41ea-94e2-03d3ada00909" containerName="limitador" containerID="cri-o://d755e72d3a1cb56fa82ba8cfecce7ef0a62b4fe5dc655b52b261562925c9edcf" gracePeriod=30 Jul 02 19:16:41.402899 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:41.402875 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-tktk7" Jul 02 19:16:41.478090 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:41.478019 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gm4ls\" (UniqueName: \"kubernetes.io/projected/a0053e6d-76a7-41ea-94e2-03d3ada00909-kube-api-access-gm4ls\") pod \"a0053e6d-76a7-41ea-94e2-03d3ada00909\" (UID: \"a0053e6d-76a7-41ea-94e2-03d3ada00909\") " Jul 02 19:16:41.478090 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:41.478084 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a0053e6d-76a7-41ea-94e2-03d3ada00909-config-file\") pod \"a0053e6d-76a7-41ea-94e2-03d3ada00909\" (UID: \"a0053e6d-76a7-41ea-94e2-03d3ada00909\") " Jul 02 19:16:41.478373 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:41.478351 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0053e6d-76a7-41ea-94e2-03d3ada00909-config-file" (OuterVolumeSpecName: "config-file") pod "a0053e6d-76a7-41ea-94e2-03d3ada00909" (UID: "a0053e6d-76a7-41ea-94e2-03d3ada00909"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:16:41.480173 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:41.480151 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0053e6d-76a7-41ea-94e2-03d3ada00909-kube-api-access-gm4ls" (OuterVolumeSpecName: "kube-api-access-gm4ls") pod "a0053e6d-76a7-41ea-94e2-03d3ada00909" (UID: "a0053e6d-76a7-41ea-94e2-03d3ada00909"). InnerVolumeSpecName "kube-api-access-gm4ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:16:41.578989 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:41.578961 2562 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a0053e6d-76a7-41ea-94e2-03d3ada00909-config-file\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:16:41.578989 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:41.578989 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm4ls\" (UniqueName: \"kubernetes.io/projected/a0053e6d-76a7-41ea-94e2-03d3ada00909-kube-api-access-gm4ls\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:16:42.169203 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:42.169171 2562 generic.go:356] "Generic (PLEG): container finished" podID="a0053e6d-76a7-41ea-94e2-03d3ada00909" containerID="d755e72d3a1cb56fa82ba8cfecce7ef0a62b4fe5dc655b52b261562925c9edcf" exitCode=0 Jul 02 19:16:42.169369 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:42.169213 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-tktk7" Jul 02 19:16:42.169369 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:42.169217 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-tktk7" event={"ID":"a0053e6d-76a7-41ea-94e2-03d3ada00909","Type":"ContainerDied","Data":"d755e72d3a1cb56fa82ba8cfecce7ef0a62b4fe5dc655b52b261562925c9edcf"} Jul 02 19:16:42.169369 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:42.169349 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-tktk7" event={"ID":"a0053e6d-76a7-41ea-94e2-03d3ada00909","Type":"ContainerDied","Data":"8bc86ea7e897981e087318fbf884055cab8dd55d4b90d17bdc4ca85477a329f6"} Jul 02 19:16:42.169473 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:42.169381 2562 scope.go:117] "RemoveContainer" containerID="d755e72d3a1cb56fa82ba8cfecce7ef0a62b4fe5dc655b52b261562925c9edcf" Jul 02 19:16:42.178298 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:42.178281 2562 scope.go:117] "RemoveContainer" containerID="d755e72d3a1cb56fa82ba8cfecce7ef0a62b4fe5dc655b52b261562925c9edcf" Jul 02 19:16:42.178549 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:16:42.178531 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d755e72d3a1cb56fa82ba8cfecce7ef0a62b4fe5dc655b52b261562925c9edcf\": container with ID starting with d755e72d3a1cb56fa82ba8cfecce7ef0a62b4fe5dc655b52b261562925c9edcf not found: ID does not exist" containerID="d755e72d3a1cb56fa82ba8cfecce7ef0a62b4fe5dc655b52b261562925c9edcf" Jul 02 19:16:42.178616 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:42.178557 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d755e72d3a1cb56fa82ba8cfecce7ef0a62b4fe5dc655b52b261562925c9edcf"} err="failed to get container status \"d755e72d3a1cb56fa82ba8cfecce7ef0a62b4fe5dc655b52b261562925c9edcf\": rpc error: code = NotFound desc = could not find container \"d755e72d3a1cb56fa82ba8cfecce7ef0a62b4fe5dc655b52b261562925c9edcf\": container with ID starting with d755e72d3a1cb56fa82ba8cfecce7ef0a62b4fe5dc655b52b261562925c9edcf not found: ID does not exist" Jul 02 19:16:42.189843 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:42.189822 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tktk7"] Jul 02 19:16:42.192322 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:42.192302 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tktk7"] Jul 02 19:16:42.777566 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:42.777533 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0053e6d-76a7-41ea-94e2-03d3ada00909" path="/var/lib/kubelet/pods/a0053e6d-76a7-41ea-94e2-03d3ada00909/volumes" Jul 02 19:16:45.830728 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:45.830696 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-bsbw7"] Jul 02 19:16:45.831228 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:45.831209 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0053e6d-76a7-41ea-94e2-03d3ada00909" containerName="limitador" Jul 02 19:16:45.831307 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:45.831232 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0053e6d-76a7-41ea-94e2-03d3ada00909" containerName="limitador" Jul 02 19:16:45.831365 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:45.831344 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0053e6d-76a7-41ea-94e2-03d3ada00909" containerName="limitador" Jul 02 19:16:45.834584 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:45.834562 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-bsbw7" Jul 02 19:16:45.837194 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:45.837176 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 02 19:16:45.837315 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:45.837237 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kg7lp\"" Jul 02 19:16:45.842490 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:45.842472 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-bsbw7"] Jul 02 19:16:45.867801 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:45.867778 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:16:45.914109 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:45.914080 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6nd9\" (UniqueName: \"kubernetes.io/projected/bb335b6a-1323-40e5-b2d6-4a426c678ef2-kube-api-access-r6nd9\") pod \"postgres-69fb4594cf-bsbw7\" (UID: \"bb335b6a-1323-40e5-b2d6-4a426c678ef2\") " pod="opendatahub/postgres-69fb4594cf-bsbw7" Jul 02 19:16:45.914240 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:45.914136 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bb335b6a-1323-40e5-b2d6-4a426c678ef2-data\") pod \"postgres-69fb4594cf-bsbw7\" (UID: \"bb335b6a-1323-40e5-b2d6-4a426c678ef2\") " pod="opendatahub/postgres-69fb4594cf-bsbw7" Jul 02 19:16:46.014888 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:46.014862 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6nd9\" (UniqueName: \"kubernetes.io/projected/bb335b6a-1323-40e5-b2d6-4a426c678ef2-kube-api-access-r6nd9\") pod \"postgres-69fb4594cf-bsbw7\" (UID: \"bb335b6a-1323-40e5-b2d6-4a426c678ef2\") " pod="opendatahub/postgres-69fb4594cf-bsbw7" Jul 02 19:16:46.015050 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:46.014909 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bb335b6a-1323-40e5-b2d6-4a426c678ef2-data\") pod \"postgres-69fb4594cf-bsbw7\" (UID: \"bb335b6a-1323-40e5-b2d6-4a426c678ef2\") " pod="opendatahub/postgres-69fb4594cf-bsbw7" Jul 02 19:16:46.015378 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:46.015327 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bb335b6a-1323-40e5-b2d6-4a426c678ef2-data\") pod \"postgres-69fb4594cf-bsbw7\" (UID: \"bb335b6a-1323-40e5-b2d6-4a426c678ef2\") " pod="opendatahub/postgres-69fb4594cf-bsbw7" Jul 02 19:16:46.022777 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:46.022751 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6nd9\" (UniqueName: \"kubernetes.io/projected/bb335b6a-1323-40e5-b2d6-4a426c678ef2-kube-api-access-r6nd9\") pod \"postgres-69fb4594cf-bsbw7\" (UID: \"bb335b6a-1323-40e5-b2d6-4a426c678ef2\") " pod="opendatahub/postgres-69fb4594cf-bsbw7" Jul 02 19:16:46.146801 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:46.146744 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-bsbw7" Jul 02 19:16:46.276261 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:46.276238 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-bsbw7"] Jul 02 19:16:46.278337 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:16:46.278309 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb335b6a_1323_40e5_b2d6_4a426c678ef2.slice/crio-3aeb1b1a9807ffdb8397ddbc14486526ae8975ae676a5ea11519fa157d6ce58c WatchSource:0}: Error finding container 3aeb1b1a9807ffdb8397ddbc14486526ae8975ae676a5ea11519fa157d6ce58c: Status 404 returned error can't find the container with id 3aeb1b1a9807ffdb8397ddbc14486526ae8975ae676a5ea11519fa157d6ce58c Jul 02 19:16:47.198094 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:47.198052 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-bsbw7" event={"ID":"bb335b6a-1323-40e5-b2d6-4a426c678ef2","Type":"ContainerStarted","Data":"3aeb1b1a9807ffdb8397ddbc14486526ae8975ae676a5ea11519fa157d6ce58c"} Jul 02 19:16:51.222455 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:51.222413 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-bsbw7" event={"ID":"bb335b6a-1323-40e5-b2d6-4a426c678ef2","Type":"ContainerStarted","Data":"b65f3d97bf70fe6ccf12641cc27a9313b42662e411bb8ee1e6037fb18a7cb121"} Jul 02 19:16:51.222822 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:51.222552 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-bsbw7" Jul 02 19:16:51.249251 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:51.249183 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-bsbw7" podStartSLOduration=1.467753201 podStartE2EDuration="6.249166831s" totalImagesPullingTime="4.78141363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:16:51.24594476 +0000 UTC m=+519.022962346" watchObservedRunningTime="2026-07-02 19:16:51.249166831 +0000 UTC m=+519.026184399" Jul 02 19:16:57.258817 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:57.258786 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-bsbw7" Jul 02 19:16:57.599138 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:57.599108 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:16:57.861087 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:16:57.861011 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:17:03.920741 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:03.920653 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:17:03.929949 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:03.929921 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-69f85d489c-fh6nx"] Jul 02 19:17:03.932757 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:03.932739 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69f85d489c-fh6nx" Jul 02 19:17:03.935362 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:03.935342 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 02 19:17:03.935479 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:03.935368 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 02 19:17:03.935667 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:03.935651 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-9kkq8\"" Jul 02 19:17:03.948013 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:03.947991 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69f85d489c-fh6nx"] Jul 02 19:17:04.018108 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:04.018056 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:17:04.077722 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:04.077693 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:17:04.080931 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:04.080900 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3dfd2d4-f671-4fe1-9d28-ede2cff05b60-webhook-cert\") pod \"maas-controller-69f85d489c-fh6nx\" (UID: \"f3dfd2d4-f671-4fe1-9d28-ede2cff05b60\") " pod="opendatahub/maas-controller-69f85d489c-fh6nx" Jul 02 19:17:04.081033 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:04.080953 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5dwc\" (UniqueName: \"kubernetes.io/projected/f3dfd2d4-f671-4fe1-9d28-ede2cff05b60-kube-api-access-m5dwc\") pod \"maas-controller-69f85d489c-fh6nx\" (UID: \"f3dfd2d4-f671-4fe1-9d28-ede2cff05b60\") " pod="opendatahub/maas-controller-69f85d489c-fh6nx" Jul 02 19:17:04.104933 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:04.104903 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c8858bc75-q4wjb"] Jul 02 19:17:04.107570 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:04.107514 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8858bc75-q4wjb" Jul 02 19:17:04.117018 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:04.116995 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c8858bc75-q4wjb"] Jul 02 19:17:04.181482 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:04.181418 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3dfd2d4-f671-4fe1-9d28-ede2cff05b60-webhook-cert\") pod \"maas-controller-69f85d489c-fh6nx\" (UID: \"f3dfd2d4-f671-4fe1-9d28-ede2cff05b60\") " pod="opendatahub/maas-controller-69f85d489c-fh6nx" Jul 02 19:17:04.181482 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:04.181461 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5dwc\" (UniqueName: \"kubernetes.io/projected/f3dfd2d4-f671-4fe1-9d28-ede2cff05b60-kube-api-access-m5dwc\") pod \"maas-controller-69f85d489c-fh6nx\" (UID: \"f3dfd2d4-f671-4fe1-9d28-ede2cff05b60\") " pod="opendatahub/maas-controller-69f85d489c-fh6nx" Jul 02 19:17:04.183962 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:04.183936 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3dfd2d4-f671-4fe1-9d28-ede2cff05b60-webhook-cert\") pod \"maas-controller-69f85d489c-fh6nx\" (UID: \"f3dfd2d4-f671-4fe1-9d28-ede2cff05b60\") " pod="opendatahub/maas-controller-69f85d489c-fh6nx" Jul 02 19:17:04.191214 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:04.191189 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5dwc\" (UniqueName: \"kubernetes.io/projected/f3dfd2d4-f671-4fe1-9d28-ede2cff05b60-kube-api-access-m5dwc\") pod \"maas-controller-69f85d489c-fh6nx\" (UID: \"f3dfd2d4-f671-4fe1-9d28-ede2cff05b60\") " pod="opendatahub/maas-controller-69f85d489c-fh6nx" Jul 02 19:17:04.243247 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:04.243220 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69f85d489c-fh6nx" Jul 02 19:17:04.282787 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:04.282761 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4c7p\" (UniqueName: \"kubernetes.io/projected/223f4ba1-8ccb-46d7-9606-59f8ee798577-kube-api-access-w4c7p\") pod \"maas-controller-c8858bc75-q4wjb\" (UID: \"223f4ba1-8ccb-46d7-9606-59f8ee798577\") " pod="opendatahub/maas-controller-c8858bc75-q4wjb" Jul 02 19:17:04.282946 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:04.282923 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/223f4ba1-8ccb-46d7-9606-59f8ee798577-webhook-cert\") pod \"maas-controller-c8858bc75-q4wjb\" (UID: \"223f4ba1-8ccb-46d7-9606-59f8ee798577\") " pod="opendatahub/maas-controller-c8858bc75-q4wjb" Jul 02 19:17:04.377672 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:04.377649 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69f85d489c-fh6nx"] Jul 02 19:17:04.379139 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:17:04.379111 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3dfd2d4_f671_4fe1_9d28_ede2cff05b60.slice/crio-4e25ed5ba4ae9f8af4854f262d7b7d8e76cf24f14c57b01f4c599d051c22e2aa WatchSource:0}: Error finding container 4e25ed5ba4ae9f8af4854f262d7b7d8e76cf24f14c57b01f4c599d051c22e2aa: Status 404 returned error can't find the container with id 4e25ed5ba4ae9f8af4854f262d7b7d8e76cf24f14c57b01f4c599d051c22e2aa Jul 02 19:17:04.384223 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:04.384200 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/223f4ba1-8ccb-46d7-9606-59f8ee798577-webhook-cert\") pod \"maas-controller-c8858bc75-q4wjb\" (UID: \"223f4ba1-8ccb-46d7-9606-59f8ee798577\") " pod="opendatahub/maas-controller-c8858bc75-q4wjb" Jul 02 19:17:04.384310 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:04.384240 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4c7p\" (UniqueName: \"kubernetes.io/projected/223f4ba1-8ccb-46d7-9606-59f8ee798577-kube-api-access-w4c7p\") pod \"maas-controller-c8858bc75-q4wjb\" (UID: \"223f4ba1-8ccb-46d7-9606-59f8ee798577\") " pod="opendatahub/maas-controller-c8858bc75-q4wjb" Jul 02 19:17:04.386998 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:04.386977 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/223f4ba1-8ccb-46d7-9606-59f8ee798577-webhook-cert\") pod \"maas-controller-c8858bc75-q4wjb\" (UID: \"223f4ba1-8ccb-46d7-9606-59f8ee798577\") " pod="opendatahub/maas-controller-c8858bc75-q4wjb" Jul 02 19:17:04.391635 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:04.391616 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4c7p\" (UniqueName: \"kubernetes.io/projected/223f4ba1-8ccb-46d7-9606-59f8ee798577-kube-api-access-w4c7p\") pod \"maas-controller-c8858bc75-q4wjb\" (UID: \"223f4ba1-8ccb-46d7-9606-59f8ee798577\") " pod="opendatahub/maas-controller-c8858bc75-q4wjb" Jul 02 19:17:04.417708 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:04.417686 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8858bc75-q4wjb" Jul 02 19:17:04.545765 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:04.545663 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c8858bc75-q4wjb"] Jul 02 19:17:04.548402 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:17:04.548369 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223f4ba1_8ccb_46d7_9606_59f8ee798577.slice/crio-74197fcc9c8f9bb7c9a57b05ea5e00eec837e7690a5203999139105ee4c9f5ed WatchSource:0}: Error finding container 74197fcc9c8f9bb7c9a57b05ea5e00eec837e7690a5203999139105ee4c9f5ed: Status 404 returned error can't find the container with id 74197fcc9c8f9bb7c9a57b05ea5e00eec837e7690a5203999139105ee4c9f5ed Jul 02 19:17:05.295780 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:05.295723 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8858bc75-q4wjb" event={"ID":"223f4ba1-8ccb-46d7-9606-59f8ee798577","Type":"ContainerStarted","Data":"74197fcc9c8f9bb7c9a57b05ea5e00eec837e7690a5203999139105ee4c9f5ed"} Jul 02 19:17:05.297223 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:05.297193 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69f85d489c-fh6nx" event={"ID":"f3dfd2d4-f671-4fe1-9d28-ede2cff05b60","Type":"ContainerStarted","Data":"4e25ed5ba4ae9f8af4854f262d7b7d8e76cf24f14c57b01f4c599d051c22e2aa"} Jul 02 19:17:06.910673 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:06.910632 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:17:07.005630 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:07.005521 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:17:07.317509 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:07.317471 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8858bc75-q4wjb" event={"ID":"223f4ba1-8ccb-46d7-9606-59f8ee798577","Type":"ContainerStarted","Data":"46be2dd4c0a43f023430ed5cdb0df1096e184ea2f78b73c22f1a78e5f58e7d89"} Jul 02 19:17:07.317711 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:07.317582 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c8858bc75-q4wjb" Jul 02 19:17:07.319069 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:07.319046 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69f85d489c-fh6nx" event={"ID":"f3dfd2d4-f671-4fe1-9d28-ede2cff05b60","Type":"ContainerStarted","Data":"dda95cfb163b2deea5670c3f37301905467f9e376885f5590685efd61920da21"} Jul 02 19:17:07.319202 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:07.319189 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-69f85d489c-fh6nx" Jul 02 19:17:07.336672 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:07.336618 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c8858bc75-q4wjb" podStartSLOduration=1.195864807 podStartE2EDuration="3.336580165s" totalImagesPullingTime="2.140715358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:17:07.33608867 +0000 UTC m=+535.113106260" watchObservedRunningTime="2026-07-02 19:17:07.336580165 +0000 UTC m=+535.113597732" Jul 02 19:17:07.354742 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:07.354699 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-69f85d489c-fh6nx" podStartSLOduration=2.034107144 podStartE2EDuration="4.354686672s" totalImagesPullingTime="2.320579528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:17:07.353078328 +0000 UTC m=+535.130095896" watchObservedRunningTime="2026-07-02 19:17:07.354686672 +0000 UTC m=+535.131704238" Jul 02 19:17:18.328584 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:18.328550 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c8858bc75-q4wjb" Jul 02 19:17:18.329052 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:18.328906 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-69f85d489c-fh6nx" Jul 02 19:17:18.391304 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:18.391277 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-69f85d489c-fh6nx"] Jul 02 19:17:18.391496 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:18.391463 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-69f85d489c-fh6nx" podUID="f3dfd2d4-f671-4fe1-9d28-ede2cff05b60" containerName="manager" containerID="cri-o://dda95cfb163b2deea5670c3f37301905467f9e376885f5590685efd61920da21" gracePeriod=30 Jul 02 19:17:18.629151 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:18.629127 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69f85d489c-fh6nx" Jul 02 19:17:18.712367 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:18.712339 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3dfd2d4-f671-4fe1-9d28-ede2cff05b60-webhook-cert\") pod \"f3dfd2d4-f671-4fe1-9d28-ede2cff05b60\" (UID: \"f3dfd2d4-f671-4fe1-9d28-ede2cff05b60\") " Jul 02 19:17:18.712549 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:18.712455 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5dwc\" (UniqueName: \"kubernetes.io/projected/f3dfd2d4-f671-4fe1-9d28-ede2cff05b60-kube-api-access-m5dwc\") pod \"f3dfd2d4-f671-4fe1-9d28-ede2cff05b60\" (UID: \"f3dfd2d4-f671-4fe1-9d28-ede2cff05b60\") " Jul 02 19:17:18.714675 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:18.714643 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3dfd2d4-f671-4fe1-9d28-ede2cff05b60-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f3dfd2d4-f671-4fe1-9d28-ede2cff05b60" (UID: "f3dfd2d4-f671-4fe1-9d28-ede2cff05b60"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:17:18.714776 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:18.714724 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3dfd2d4-f671-4fe1-9d28-ede2cff05b60-kube-api-access-m5dwc" (OuterVolumeSpecName: "kube-api-access-m5dwc") pod "f3dfd2d4-f671-4fe1-9d28-ede2cff05b60" (UID: "f3dfd2d4-f671-4fe1-9d28-ede2cff05b60"). InnerVolumeSpecName "kube-api-access-m5dwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:17:18.813670 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:18.813641 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5dwc\" (UniqueName: \"kubernetes.io/projected/f3dfd2d4-f671-4fe1-9d28-ede2cff05b60-kube-api-access-m5dwc\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:17:18.813670 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:18.813670 2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3dfd2d4-f671-4fe1-9d28-ede2cff05b60-webhook-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:17:19.376952 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:19.376919 2562 generic.go:356] "Generic (PLEG): container finished" podID="f3dfd2d4-f671-4fe1-9d28-ede2cff05b60" containerID="dda95cfb163b2deea5670c3f37301905467f9e376885f5590685efd61920da21" exitCode=0 Jul 02 19:17:19.377345 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:19.376968 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69f85d489c-fh6nx" Jul 02 19:17:19.377345 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:19.377005 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69f85d489c-fh6nx" event={"ID":"f3dfd2d4-f671-4fe1-9d28-ede2cff05b60","Type":"ContainerDied","Data":"dda95cfb163b2deea5670c3f37301905467f9e376885f5590685efd61920da21"} Jul 02 19:17:19.377345 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:19.377060 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69f85d489c-fh6nx" event={"ID":"f3dfd2d4-f671-4fe1-9d28-ede2cff05b60","Type":"ContainerDied","Data":"4e25ed5ba4ae9f8af4854f262d7b7d8e76cf24f14c57b01f4c599d051c22e2aa"} Jul 02 19:17:19.377345 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:19.377083 2562 scope.go:117] "RemoveContainer" containerID="dda95cfb163b2deea5670c3f37301905467f9e376885f5590685efd61920da21" Jul 02 19:17:19.385666 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:19.385649 2562 scope.go:117] "RemoveContainer" containerID="dda95cfb163b2deea5670c3f37301905467f9e376885f5590685efd61920da21" Jul 02 19:17:19.385927 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:17:19.385907 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dda95cfb163b2deea5670c3f37301905467f9e376885f5590685efd61920da21\": container with ID starting with dda95cfb163b2deea5670c3f37301905467f9e376885f5590685efd61920da21 not found: ID does not exist" containerID="dda95cfb163b2deea5670c3f37301905467f9e376885f5590685efd61920da21" Jul 02 19:17:19.386022 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:19.385934 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda95cfb163b2deea5670c3f37301905467f9e376885f5590685efd61920da21"} err="failed to get container status \"dda95cfb163b2deea5670c3f37301905467f9e376885f5590685efd61920da21\": rpc error: code = NotFound desc = could not find container \"dda95cfb163b2deea5670c3f37301905467f9e376885f5590685efd61920da21\": container with ID starting with dda95cfb163b2deea5670c3f37301905467f9e376885f5590685efd61920da21 not found: ID does not exist" Jul 02 19:17:19.393707 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:19.393688 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-69f85d489c-fh6nx"] Jul 02 19:17:19.397664 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:19.397646 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-69f85d489c-fh6nx"] Jul 02 19:17:20.777265 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:20.777234 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3dfd2d4-f671-4fe1-9d28-ede2cff05b60" path="/var/lib/kubelet/pods/f3dfd2d4-f671-4fe1-9d28-ede2cff05b60/volumes" Jul 02 19:17:38.211865 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:38.211828 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:17:38.617051 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:38.617011 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:17:38.661669 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:38.661640 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-57b5579877-bg8h6"] Jul 02 19:17:38.662011 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:38.662000 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3dfd2d4-f671-4fe1-9d28-ede2cff05b60" containerName="manager" Jul 02 19:17:38.662056 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:38.662014 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3dfd2d4-f671-4fe1-9d28-ede2cff05b60" containerName="manager" Jul 02 19:17:38.662089 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:38.662079 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3dfd2d4-f671-4fe1-9d28-ede2cff05b60" containerName="manager" Jul 02 19:17:38.664293 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:38.664268 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-57b5579877-bg8h6" Jul 02 19:17:38.667346 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:38.667299 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-rtxc5\"" Jul 02 19:17:38.668398 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:38.668377 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 02 19:17:38.681199 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:38.681178 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-57b5579877-bg8h6"] Jul 02 19:17:38.702022 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:38.701996 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:17:38.781569 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:38.781538 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d41ed33-5de8-4d2a-86bd-3815fc777516-maas-api-tls\") pod \"maas-api-57b5579877-bg8h6\" (UID: \"6d41ed33-5de8-4d2a-86bd-3815fc777516\") " pod="opendatahub/maas-api-57b5579877-bg8h6" Jul 02 19:17:38.781729 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:38.781621 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzvhx\" (UniqueName: \"kubernetes.io/projected/6d41ed33-5de8-4d2a-86bd-3815fc777516-kube-api-access-dzvhx\") pod \"maas-api-57b5579877-bg8h6\" (UID: \"6d41ed33-5de8-4d2a-86bd-3815fc777516\") " pod="opendatahub/maas-api-57b5579877-bg8h6" Jul 02 19:17:38.781729 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:38.781670 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d41ed33-5de8-4d2a-86bd-3815fc777516-openshift-service-ca\") pod \"maas-api-57b5579877-bg8h6\" (UID: \"6d41ed33-5de8-4d2a-86bd-3815fc777516\") " pod="opendatahub/maas-api-57b5579877-bg8h6" Jul 02 19:17:38.882940 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:38.882861 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzvhx\" (UniqueName: \"kubernetes.io/projected/6d41ed33-5de8-4d2a-86bd-3815fc777516-kube-api-access-dzvhx\") pod \"maas-api-57b5579877-bg8h6\" (UID: \"6d41ed33-5de8-4d2a-86bd-3815fc777516\") " pod="opendatahub/maas-api-57b5579877-bg8h6" Jul 02 19:17:38.882940 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:38.882906 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d41ed33-5de8-4d2a-86bd-3815fc777516-openshift-service-ca\") pod \"maas-api-57b5579877-bg8h6\" (UID: \"6d41ed33-5de8-4d2a-86bd-3815fc777516\") " pod="opendatahub/maas-api-57b5579877-bg8h6" Jul 02 19:17:38.883106 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:38.882972 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d41ed33-5de8-4d2a-86bd-3815fc777516-maas-api-tls\") pod \"maas-api-57b5579877-bg8h6\" (UID: \"6d41ed33-5de8-4d2a-86bd-3815fc777516\") " pod="opendatahub/maas-api-57b5579877-bg8h6" Jul 02 19:17:38.883746 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:38.883709 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d41ed33-5de8-4d2a-86bd-3815fc777516-openshift-service-ca\") pod \"maas-api-57b5579877-bg8h6\" (UID: \"6d41ed33-5de8-4d2a-86bd-3815fc777516\") " pod="opendatahub/maas-api-57b5579877-bg8h6" Jul 02 19:17:38.885715 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:38.885692 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d41ed33-5de8-4d2a-86bd-3815fc777516-maas-api-tls\") pod \"maas-api-57b5579877-bg8h6\" (UID: \"6d41ed33-5de8-4d2a-86bd-3815fc777516\") " pod="opendatahub/maas-api-57b5579877-bg8h6" Jul 02 19:17:38.891183 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:38.891161 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzvhx\" (UniqueName: \"kubernetes.io/projected/6d41ed33-5de8-4d2a-86bd-3815fc777516-kube-api-access-dzvhx\") pod \"maas-api-57b5579877-bg8h6\" (UID: \"6d41ed33-5de8-4d2a-86bd-3815fc777516\") " pod="opendatahub/maas-api-57b5579877-bg8h6" Jul 02 19:17:38.975200 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:38.975171 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-57b5579877-bg8h6" Jul 02 19:17:39.052686 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.052653 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:17:39.122516 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.122489 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-57b5579877-bg8h6"] Jul 02 19:17:39.125073 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:17:39.125044 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d41ed33_5de8_4d2a_86bd_3815fc777516.slice/crio-04bab54833960c47666a738657256682a2f36685a18651527b90b83d3fde92bd WatchSource:0}: Error finding container 04bab54833960c47666a738657256682a2f36685a18651527b90b83d3fde92bd: Status 404 returned error can't find the container with id 04bab54833960c47666a738657256682a2f36685a18651527b90b83d3fde92bd Jul 02 19:17:39.125501 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.125477 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:17:39.136337 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.136282 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-qp4rb"] Jul 02 19:17:39.139693 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.139674 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-qp4rb" Jul 02 19:17:39.142461 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.142443 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 02 19:17:39.151172 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.151117 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-qp4rb"] Jul 02 19:17:39.184894 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.184866 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-889xr"] Jul 02 19:17:39.187540 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.187525 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-889xr" Jul 02 19:17:39.190348 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.190330 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-fqg6f\"" Jul 02 19:17:39.199801 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.199779 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-889xr"] Jul 02 19:17:39.286817 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.286783 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8a5510-9e86-44dc-98ab-600383f4b571-plugins-config-volume\") pod \"payload-pre-processing-7964457696-qp4rb\" (UID: \"5c8a5510-9e86-44dc-98ab-600383f4b571\") " pod="openshift-ingress/payload-pre-processing-7964457696-qp4rb" Jul 02 19:17:39.287237 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.286874 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctkrh\" (UniqueName: \"kubernetes.io/projected/5c8a5510-9e86-44dc-98ab-600383f4b571-kube-api-access-ctkrh\") pod \"payload-pre-processing-7964457696-qp4rb\" (UID: \"5c8a5510-9e86-44dc-98ab-600383f4b571\") " pod="openshift-ingress/payload-pre-processing-7964457696-qp4rb" Jul 02 19:17:39.287237 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.287000 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78h4z\" (UniqueName: \"kubernetes.io/projected/748bb359-fd6a-4f3a-91e9-c2fe8e47d941-kube-api-access-78h4z\") pod \"payload-processing-58f86c7798-889xr\" (UID: \"748bb359-fd6a-4f3a-91e9-c2fe8e47d941\") " pod="openshift-ingress/payload-processing-58f86c7798-889xr" Jul 02 19:17:39.287237 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.287113 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/748bb359-fd6a-4f3a-91e9-c2fe8e47d941-plugins-config-volume\") pod \"payload-processing-58f86c7798-889xr\" (UID: \"748bb359-fd6a-4f3a-91e9-c2fe8e47d941\") " pod="openshift-ingress/payload-processing-58f86c7798-889xr" Jul 02 19:17:39.387925 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.387847 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78h4z\" (UniqueName: \"kubernetes.io/projected/748bb359-fd6a-4f3a-91e9-c2fe8e47d941-kube-api-access-78h4z\") pod \"payload-processing-58f86c7798-889xr\" (UID: \"748bb359-fd6a-4f3a-91e9-c2fe8e47d941\") " pod="openshift-ingress/payload-processing-58f86c7798-889xr" Jul 02 19:17:39.387925 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.387907 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/748bb359-fd6a-4f3a-91e9-c2fe8e47d941-plugins-config-volume\") pod \"payload-processing-58f86c7798-889xr\" (UID: \"748bb359-fd6a-4f3a-91e9-c2fe8e47d941\") " pod="openshift-ingress/payload-processing-58f86c7798-889xr" Jul 02 19:17:39.388102 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.387932 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8a5510-9e86-44dc-98ab-600383f4b571-plugins-config-volume\") pod \"payload-pre-processing-7964457696-qp4rb\" (UID: \"5c8a5510-9e86-44dc-98ab-600383f4b571\") " pod="openshift-ingress/payload-pre-processing-7964457696-qp4rb" Jul 02 19:17:39.388102 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.387973 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctkrh\" (UniqueName: \"kubernetes.io/projected/5c8a5510-9e86-44dc-98ab-600383f4b571-kube-api-access-ctkrh\") pod \"payload-pre-processing-7964457696-qp4rb\" (UID: \"5c8a5510-9e86-44dc-98ab-600383f4b571\") " pod="openshift-ingress/payload-pre-processing-7964457696-qp4rb" Jul 02 19:17:39.388658 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.388637 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/748bb359-fd6a-4f3a-91e9-c2fe8e47d941-plugins-config-volume\") pod \"payload-processing-58f86c7798-889xr\" (UID: \"748bb359-fd6a-4f3a-91e9-c2fe8e47d941\") " pod="openshift-ingress/payload-processing-58f86c7798-889xr" Jul 02 19:17:39.388751 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.388734 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8a5510-9e86-44dc-98ab-600383f4b571-plugins-config-volume\") pod \"payload-pre-processing-7964457696-qp4rb\" (UID: \"5c8a5510-9e86-44dc-98ab-600383f4b571\") " pod="openshift-ingress/payload-pre-processing-7964457696-qp4rb" Jul 02 19:17:39.396047 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.396024 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78h4z\" (UniqueName: \"kubernetes.io/projected/748bb359-fd6a-4f3a-91e9-c2fe8e47d941-kube-api-access-78h4z\") pod \"payload-processing-58f86c7798-889xr\" (UID: \"748bb359-fd6a-4f3a-91e9-c2fe8e47d941\") " pod="openshift-ingress/payload-processing-58f86c7798-889xr" Jul 02 19:17:39.396144 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.396025 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctkrh\" (UniqueName: \"kubernetes.io/projected/5c8a5510-9e86-44dc-98ab-600383f4b571-kube-api-access-ctkrh\") pod \"payload-pre-processing-7964457696-qp4rb\" (UID: \"5c8a5510-9e86-44dc-98ab-600383f4b571\") " pod="openshift-ingress/payload-pre-processing-7964457696-qp4rb" Jul 02 19:17:39.451679 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.451652 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-qp4rb" Jul 02 19:17:39.471980 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.471950 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-57b5579877-bg8h6" event={"ID":"6d41ed33-5de8-4d2a-86bd-3815fc777516","Type":"ContainerStarted","Data":"04bab54833960c47666a738657256682a2f36685a18651527b90b83d3fde92bd"} Jul 02 19:17:39.499344 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.499315 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-889xr" Jul 02 19:17:39.598373 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.598345 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-qp4rb"] Jul 02 19:17:39.601733 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:17:39.601698 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8a5510_9e86_44dc_98ab_600383f4b571.slice/crio-ad154dea9c24d27993c237fa49db075549b1f67330a7dd4bc3d44d697054d111 WatchSource:0}: Error finding container ad154dea9c24d27993c237fa49db075549b1f67330a7dd4bc3d44d697054d111: Status 404 returned error can't find the container with id ad154dea9c24d27993c237fa49db075549b1f67330a7dd4bc3d44d697054d111 Jul 02 19:17:39.643957 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:39.643880 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-889xr"] Jul 02 19:17:39.646536 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:17:39.646509 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod748bb359_fd6a_4f3a_91e9_c2fe8e47d941.slice/crio-7ef96109e2f17d12cbdb54001320f3f3bdab57428ae10b813b8dcb359d1a7aa3 WatchSource:0}: Error finding container 7ef96109e2f17d12cbdb54001320f3f3bdab57428ae10b813b8dcb359d1a7aa3: Status 404 returned error can't find the container with id 7ef96109e2f17d12cbdb54001320f3f3bdab57428ae10b813b8dcb359d1a7aa3 Jul 02 19:17:40.481216 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:40.481176 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-qp4rb" event={"ID":"5c8a5510-9e86-44dc-98ab-600383f4b571","Type":"ContainerStarted","Data":"ad154dea9c24d27993c237fa49db075549b1f67330a7dd4bc3d44d697054d111"} Jul 02 19:17:40.483274 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:40.483232 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-889xr" event={"ID":"748bb359-fd6a-4f3a-91e9-c2fe8e47d941","Type":"ContainerStarted","Data":"7ef96109e2f17d12cbdb54001320f3f3bdab57428ae10b813b8dcb359d1a7aa3"} Jul 02 19:17:42.498834 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:42.498795 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-889xr" event={"ID":"748bb359-fd6a-4f3a-91e9-c2fe8e47d941","Type":"ContainerStarted","Data":"45b121aa0a69120411134ec71c76706129877c69b936791b113c4a192d7d54c5"} Jul 02 19:17:42.499270 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:42.498922 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-889xr" Jul 02 19:17:42.500419 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:42.500389 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-qp4rb" event={"ID":"5c8a5510-9e86-44dc-98ab-600383f4b571","Type":"ContainerStarted","Data":"748a89299429ef644b49b6fcc95296282b820abb547e8237d49cf8918f0e166b"} Jul 02 19:17:42.500542 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:42.500444 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-qp4rb" Jul 02 19:17:42.501778 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:42.501758 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-57b5579877-bg8h6" event={"ID":"6d41ed33-5de8-4d2a-86bd-3815fc777516","Type":"ContainerStarted","Data":"f67253f4d757b522b2120d9bc8a3f43449dc64d9cccfe108f176f030470c46c4"} Jul 02 19:17:42.501898 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:42.501884 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-57b5579877-bg8h6" Jul 02 19:17:42.517975 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:42.517934 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-889xr" podStartSLOduration=1.305357657 podStartE2EDuration="3.517921137s" totalImagesPullingTime="2.21256348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:17:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:17:42.516529955 +0000 UTC m=+570.293547542" watchObservedRunningTime="2026-07-02 19:17:42.517921137 +0000 UTC m=+570.294938703" Jul 02 19:17:42.531145 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:42.531076 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-qp4rb" podStartSLOduration=1.274092992 podStartE2EDuration="3.531050519s" totalImagesPullingTime="2.256957527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:17:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:17:42.531051779 +0000 UTC m=+570.308069366" watchObservedRunningTime="2026-07-02 19:17:42.531050519 +0000 UTC m=+570.308068106" Jul 02 19:17:42.547458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:42.547410 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-57b5579877-bg8h6" podStartSLOduration=1.811796868 podStartE2EDuration="4.547395402s" totalImagesPullingTime="2.735598534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:17:42.547303735 +0000 UTC m=+570.324321322" watchObservedRunningTime="2026-07-02 19:17:42.547395402 +0000 UTC m=+570.324412970" Jul 02 19:17:48.511620 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:48.511567 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-57b5579877-bg8h6" Jul 02 19:17:49.553330 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:49.553302 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:17:50.627901 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:50.627858 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:17:53.508367 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:53.508339 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-889xr" Jul 02 19:17:53.508766 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:17:53.508401 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-qp4rb" Jul 02 19:18:08.989967 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:08.989924 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr"] Jul 02 19:18:08.992826 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:08.992804 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:08.997374 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:08.997350 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 02 19:18:09.011633 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.011615 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 02 19:18:09.014883 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.014814 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 02 19:18:09.015306 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.015284 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mvwcw\"" Jul 02 19:18:09.017679 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.017654 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr"] Jul 02 19:18:09.047501 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.047468 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3d9b000-85a2-4def-9a25-105a85023e15-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr\" (UID: \"b3d9b000-85a2-4def-9a25-105a85023e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:09.047663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.047503 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tk96\" (UniqueName: \"kubernetes.io/projected/b3d9b000-85a2-4def-9a25-105a85023e15-kube-api-access-9tk96\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr\" (UID: \"b3d9b000-85a2-4def-9a25-105a85023e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:09.047663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.047526 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3d9b000-85a2-4def-9a25-105a85023e15-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr\" (UID: \"b3d9b000-85a2-4def-9a25-105a85023e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:09.047663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.047577 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3d9b000-85a2-4def-9a25-105a85023e15-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr\" (UID: \"b3d9b000-85a2-4def-9a25-105a85023e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:09.047802 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.047664 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3d9b000-85a2-4def-9a25-105a85023e15-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr\" (UID: \"b3d9b000-85a2-4def-9a25-105a85023e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:09.047802 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.047723 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3d9b000-85a2-4def-9a25-105a85023e15-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr\" (UID: \"b3d9b000-85a2-4def-9a25-105a85023e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:09.054417 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.054396 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:18:09.148135 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.148107 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3d9b000-85a2-4def-9a25-105a85023e15-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr\" (UID: \"b3d9b000-85a2-4def-9a25-105a85023e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:09.148258 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.148157 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3d9b000-85a2-4def-9a25-105a85023e15-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr\" (UID: \"b3d9b000-85a2-4def-9a25-105a85023e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:09.148258 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.148203 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3d9b000-85a2-4def-9a25-105a85023e15-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr\" (UID: \"b3d9b000-85a2-4def-9a25-105a85023e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:09.148258 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.148225 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tk96\" (UniqueName: \"kubernetes.io/projected/b3d9b000-85a2-4def-9a25-105a85023e15-kube-api-access-9tk96\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr\" (UID: \"b3d9b000-85a2-4def-9a25-105a85023e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:09.148258 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.148245 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3d9b000-85a2-4def-9a25-105a85023e15-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr\" (UID: \"b3d9b000-85a2-4def-9a25-105a85023e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:09.148415 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.148264 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3d9b000-85a2-4def-9a25-105a85023e15-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr\" (UID: \"b3d9b000-85a2-4def-9a25-105a85023e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:09.148553 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.148531 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3d9b000-85a2-4def-9a25-105a85023e15-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr\" (UID: \"b3d9b000-85a2-4def-9a25-105a85023e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:09.148629 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.148586 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3d9b000-85a2-4def-9a25-105a85023e15-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr\" (UID: \"b3d9b000-85a2-4def-9a25-105a85023e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:09.148704 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.148689 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3d9b000-85a2-4def-9a25-105a85023e15-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr\" (UID: \"b3d9b000-85a2-4def-9a25-105a85023e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:09.150608 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.150570 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3d9b000-85a2-4def-9a25-105a85023e15-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr\" (UID: \"b3d9b000-85a2-4def-9a25-105a85023e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:09.150869 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.150850 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3d9b000-85a2-4def-9a25-105a85023e15-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr\" (UID: \"b3d9b000-85a2-4def-9a25-105a85023e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:09.157222 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.157198 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tk96\" (UniqueName: \"kubernetes.io/projected/b3d9b000-85a2-4def-9a25-105a85023e15-kube-api-access-9tk96\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr\" (UID: \"b3d9b000-85a2-4def-9a25-105a85023e15\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:09.304430 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.304408 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:09.443050 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.442966 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr"] Jul 02 19:18:09.446187 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:18:09.446160 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d9b000_85a2_4def_9a25_105a85023e15.slice/crio-da556b1042691dd03595a6d4db9132c4e68416b4296a72c161730a0fcb95bba2 WatchSource:0}: Error finding container da556b1042691dd03595a6d4db9132c4e68416b4296a72c161730a0fcb95bba2: Status 404 returned error can't find the container with id da556b1042691dd03595a6d4db9132c4e68416b4296a72c161730a0fcb95bba2 Jul 02 19:18:09.629734 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:09.629642 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" event={"ID":"b3d9b000-85a2-4def-9a25-105a85023e15","Type":"ContainerStarted","Data":"da556b1042691dd03595a6d4db9132c4e68416b4296a72c161730a0fcb95bba2"} Jul 02 19:18:10.636330 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:10.636299 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" event={"ID":"b3d9b000-85a2-4def-9a25-105a85023e15","Type":"ContainerStarted","Data":"8db80993975950f844623b3433a487581925ac48f7250bec6b84d6dd8acc939b"} Jul 02 19:18:10.661870 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:10.661822 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" podStartSLOduration=1.632274282 podStartE2EDuration="2.661807835s" totalImagesPullingTime="1.029533553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:18:10.65939794 +0000 UTC m=+598.436415506" watchObservedRunningTime="2026-07-02 19:18:10.661807835 +0000 UTC m=+598.438825402" Jul 02 19:18:19.304760 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:19.304726 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:19.305254 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:19.304786 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:19.317271 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:19.317248 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:19.690816 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:19.690735 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr" Jul 02 19:18:26.833748 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:26.833712 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:18:33.785788 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:33.785710 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:18:33.887020 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:33.886988 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:18:38.091958 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.091925 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l"] Jul 02 19:18:38.094865 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.094842 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:38.098399 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.098379 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 02 19:18:38.108838 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.108814 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l"] Jul 02 19:18:38.130136 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.130110 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:18:38.204252 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.204225 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/490379dc-0d79-4b3f-a1e4-35bb39dc4f5d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48l9l\" (UID: \"490379dc-0d79-4b3f-a1e4-35bb39dc4f5d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:38.204359 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.204260 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/490379dc-0d79-4b3f-a1e4-35bb39dc4f5d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48l9l\" (UID: \"490379dc-0d79-4b3f-a1e4-35bb39dc4f5d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:38.204359 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.204294 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/490379dc-0d79-4b3f-a1e4-35bb39dc4f5d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48l9l\" (UID: \"490379dc-0d79-4b3f-a1e4-35bb39dc4f5d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:38.204443 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.204373 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/490379dc-0d79-4b3f-a1e4-35bb39dc4f5d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48l9l\" (UID: \"490379dc-0d79-4b3f-a1e4-35bb39dc4f5d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:38.204443 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.204410 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrfzx\" (UniqueName: \"kubernetes.io/projected/490379dc-0d79-4b3f-a1e4-35bb39dc4f5d-kube-api-access-lrfzx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48l9l\" (UID: \"490379dc-0d79-4b3f-a1e4-35bb39dc4f5d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:38.204443 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.204432 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/490379dc-0d79-4b3f-a1e4-35bb39dc4f5d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48l9l\" (UID: \"490379dc-0d79-4b3f-a1e4-35bb39dc4f5d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:38.305152 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.305124 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/490379dc-0d79-4b3f-a1e4-35bb39dc4f5d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48l9l\" (UID: \"490379dc-0d79-4b3f-a1e4-35bb39dc4f5d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:38.305152 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.305157 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/490379dc-0d79-4b3f-a1e4-35bb39dc4f5d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48l9l\" (UID: \"490379dc-0d79-4b3f-a1e4-35bb39dc4f5d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:38.305334 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.305188 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/490379dc-0d79-4b3f-a1e4-35bb39dc4f5d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48l9l\" (UID: \"490379dc-0d79-4b3f-a1e4-35bb39dc4f5d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:38.305334 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.305207 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/490379dc-0d79-4b3f-a1e4-35bb39dc4f5d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48l9l\" (UID: \"490379dc-0d79-4b3f-a1e4-35bb39dc4f5d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:38.305334 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.305245 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrfzx\" (UniqueName: \"kubernetes.io/projected/490379dc-0d79-4b3f-a1e4-35bb39dc4f5d-kube-api-access-lrfzx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48l9l\" (UID: \"490379dc-0d79-4b3f-a1e4-35bb39dc4f5d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:38.305334 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.305266 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/490379dc-0d79-4b3f-a1e4-35bb39dc4f5d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48l9l\" (UID: \"490379dc-0d79-4b3f-a1e4-35bb39dc4f5d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:38.305575 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.305553 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/490379dc-0d79-4b3f-a1e4-35bb39dc4f5d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48l9l\" (UID: \"490379dc-0d79-4b3f-a1e4-35bb39dc4f5d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:38.305670 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.305626 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/490379dc-0d79-4b3f-a1e4-35bb39dc4f5d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48l9l\" (UID: \"490379dc-0d79-4b3f-a1e4-35bb39dc4f5d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:38.305728 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.305695 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/490379dc-0d79-4b3f-a1e4-35bb39dc4f5d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48l9l\" (UID: \"490379dc-0d79-4b3f-a1e4-35bb39dc4f5d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:38.307703 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.307685 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/490379dc-0d79-4b3f-a1e4-35bb39dc4f5d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48l9l\" (UID: \"490379dc-0d79-4b3f-a1e4-35bb39dc4f5d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:38.308062 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.308033 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/490379dc-0d79-4b3f-a1e4-35bb39dc4f5d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48l9l\" (UID: \"490379dc-0d79-4b3f-a1e4-35bb39dc4f5d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:38.314792 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.314768 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrfzx\" (UniqueName: \"kubernetes.io/projected/490379dc-0d79-4b3f-a1e4-35bb39dc4f5d-kube-api-access-lrfzx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-48l9l\" (UID: \"490379dc-0d79-4b3f-a1e4-35bb39dc4f5d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:38.405761 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.405698 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:38.543174 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.543151 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l"] Jul 02 19:18:38.544872 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:18:38.544842 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490379dc_0d79_4b3f_a1e4_35bb39dc4f5d.slice/crio-f096b259944ea69453192bf2fbc6ccdff96cc3963c5306504b3387f1b0068bf7 WatchSource:0}: Error finding container f096b259944ea69453192bf2fbc6ccdff96cc3963c5306504b3387f1b0068bf7: Status 404 returned error can't find the container with id f096b259944ea69453192bf2fbc6ccdff96cc3963c5306504b3387f1b0068bf7 Jul 02 19:18:38.546326 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.546309 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:18:38.777657 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:38.777631 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" event={"ID":"490379dc-0d79-4b3f-a1e4-35bb39dc4f5d","Type":"ContainerStarted","Data":"f096b259944ea69453192bf2fbc6ccdff96cc3963c5306504b3387f1b0068bf7"} Jul 02 19:18:39.782512 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:39.782478 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" event={"ID":"490379dc-0d79-4b3f-a1e4-35bb39dc4f5d","Type":"ContainerStarted","Data":"bb1ac273d876052cadd727578d77834c611ebe79b1a512870ce031cf087e87b9"} Jul 02 19:18:39.801265 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:39.801214 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" podStartSLOduration=1.621037069 podStartE2EDuration="1.801194951s" totalImagesPullingTime="180.157882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:18:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:18:39.800115196 +0000 UTC m=+627.577132804" watchObservedRunningTime="2026-07-02 19:18:39.801194951 +0000 UTC m=+627.578212519" Jul 02 19:18:48.406903 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:48.406862 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:48.407380 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:48.406936 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:48.419014 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:48.418992 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:48.838042 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:48.838015 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-48l9l" Jul 02 19:18:52.825531 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:18:52.825492 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:19:18.403325 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:18.403291 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:19:18.501575 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:18.501548 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:19:19.333347 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:19.333306 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:19:26.402927 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.402893 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b"] Jul 02 19:19:26.406014 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.405997 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:26.408554 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.408534 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 02 19:19:26.415505 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.415479 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b"] Jul 02 19:19:26.495775 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.495742 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:19:26.521032 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.521005 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e457b0f4-4de5-47ee-8f0f-935b4d4c81e6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkd7b\" (UID: \"e457b0f4-4de5-47ee-8f0f-935b4d4c81e6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:26.521189 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.521077 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e457b0f4-4de5-47ee-8f0f-935b4d4c81e6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkd7b\" (UID: \"e457b0f4-4de5-47ee-8f0f-935b4d4c81e6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:26.521189 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.521098 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e457b0f4-4de5-47ee-8f0f-935b4d4c81e6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkd7b\" (UID: \"e457b0f4-4de5-47ee-8f0f-935b4d4c81e6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:26.521189 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.521116 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e457b0f4-4de5-47ee-8f0f-935b4d4c81e6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkd7b\" (UID: \"e457b0f4-4de5-47ee-8f0f-935b4d4c81e6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:26.521189 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.521147 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e457b0f4-4de5-47ee-8f0f-935b4d4c81e6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkd7b\" (UID: \"e457b0f4-4de5-47ee-8f0f-935b4d4c81e6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:26.521359 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.521220 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8629\" (UniqueName: \"kubernetes.io/projected/e457b0f4-4de5-47ee-8f0f-935b4d4c81e6-kube-api-access-m8629\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkd7b\" (UID: \"e457b0f4-4de5-47ee-8f0f-935b4d4c81e6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:26.622022 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.621985 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e457b0f4-4de5-47ee-8f0f-935b4d4c81e6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkd7b\" (UID: \"e457b0f4-4de5-47ee-8f0f-935b4d4c81e6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:26.622022 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.622022 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e457b0f4-4de5-47ee-8f0f-935b4d4c81e6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkd7b\" (UID: \"e457b0f4-4de5-47ee-8f0f-935b4d4c81e6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:26.622202 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.622043 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e457b0f4-4de5-47ee-8f0f-935b4d4c81e6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkd7b\" (UID: \"e457b0f4-4de5-47ee-8f0f-935b4d4c81e6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:26.622202 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.622088 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e457b0f4-4de5-47ee-8f0f-935b4d4c81e6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkd7b\" (UID: \"e457b0f4-4de5-47ee-8f0f-935b4d4c81e6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:26.622202 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.622121 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8629\" (UniqueName: \"kubernetes.io/projected/e457b0f4-4de5-47ee-8f0f-935b4d4c81e6-kube-api-access-m8629\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkd7b\" (UID: \"e457b0f4-4de5-47ee-8f0f-935b4d4c81e6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:26.622202 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.622176 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e457b0f4-4de5-47ee-8f0f-935b4d4c81e6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkd7b\" (UID: \"e457b0f4-4de5-47ee-8f0f-935b4d4c81e6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:26.622450 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.622429 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e457b0f4-4de5-47ee-8f0f-935b4d4c81e6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkd7b\" (UID: \"e457b0f4-4de5-47ee-8f0f-935b4d4c81e6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:26.622515 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.622489 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e457b0f4-4de5-47ee-8f0f-935b4d4c81e6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkd7b\" (UID: \"e457b0f4-4de5-47ee-8f0f-935b4d4c81e6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:26.622653 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.622632 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e457b0f4-4de5-47ee-8f0f-935b4d4c81e6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkd7b\" (UID: \"e457b0f4-4de5-47ee-8f0f-935b4d4c81e6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:26.624470 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.624441 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e457b0f4-4de5-47ee-8f0f-935b4d4c81e6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkd7b\" (UID: \"e457b0f4-4de5-47ee-8f0f-935b4d4c81e6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:26.624649 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.624633 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e457b0f4-4de5-47ee-8f0f-935b4d4c81e6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkd7b\" (UID: \"e457b0f4-4de5-47ee-8f0f-935b4d4c81e6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:26.629999 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.629970 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8629\" (UniqueName: \"kubernetes.io/projected/e457b0f4-4de5-47ee-8f0f-935b4d4c81e6-kube-api-access-m8629\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lkd7b\" (UID: \"e457b0f4-4de5-47ee-8f0f-935b4d4c81e6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:26.716901 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.716843 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:26.850897 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:26.850872 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b"] Jul 02 19:19:26.852780 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:19:26.852753 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode457b0f4_4de5_47ee_8f0f_935b4d4c81e6.slice/crio-3b5fdf220cfc596e6519d4263811f1678e6162eda50149db503c96a5b15d5aec WatchSource:0}: Error finding container 3b5fdf220cfc596e6519d4263811f1678e6162eda50149db503c96a5b15d5aec: Status 404 returned error can't find the container with id 3b5fdf220cfc596e6519d4263811f1678e6162eda50149db503c96a5b15d5aec Jul 02 19:19:27.011506 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:27.011480 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" event={"ID":"e457b0f4-4de5-47ee-8f0f-935b4d4c81e6","Type":"ContainerStarted","Data":"3b5fdf220cfc596e6519d4263811f1678e6162eda50149db503c96a5b15d5aec"} Jul 02 19:19:28.018015 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:28.017984 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" event={"ID":"e457b0f4-4de5-47ee-8f0f-935b4d4c81e6","Type":"ContainerStarted","Data":"bc79503c7e3a3e5873b2baa6cc046b9114bf15fbba925b0a7c0da0b44b9ca757"} Jul 02 19:19:28.041413 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:28.041360 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" podStartSLOduration=1.88672454 podStartE2EDuration="2.041344532s" totalImagesPullingTime="154.619992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:19:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:19:28.038659591 +0000 UTC m=+675.815677172" watchObservedRunningTime="2026-07-02 19:19:28.041344532 +0000 UTC m=+675.818362098" Jul 02 19:19:34.812482 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:34.812443 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:19:36.717750 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:36.717703 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:36.718528 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:36.717794 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:36.730717 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:36.730695 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:37.074269 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:37.074238 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lkd7b" Jul 02 19:19:48.074381 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:19:48.074350 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:20:24.130824 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:20:24.130731 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:20:24.230820 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:20:24.230789 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:20:25.300896 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:20:25.300863 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:20:25.391605 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:20:25.391573 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:20:34.609681 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:20:34.609652 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:21:40.806740 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:40.806667 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:21:40.915430 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:40.915385 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c8858bc75-q4wjb"] Jul 02 19:21:40.915684 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:40.915657 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-c8858bc75-q4wjb" podUID="223f4ba1-8ccb-46d7-9606-59f8ee798577" containerName="manager" containerID="cri-o://46be2dd4c0a43f023430ed5cdb0df1096e184ea2f78b73c22f1a78e5f58e7d89" gracePeriod=30 Jul 02 19:21:41.161560 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:41.161528 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8858bc75-q4wjb" Jul 02 19:21:41.285192 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:41.285155 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4c7p\" (UniqueName: \"kubernetes.io/projected/223f4ba1-8ccb-46d7-9606-59f8ee798577-kube-api-access-w4c7p\") pod \"223f4ba1-8ccb-46d7-9606-59f8ee798577\" (UID: \"223f4ba1-8ccb-46d7-9606-59f8ee798577\") " Jul 02 19:21:41.285357 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:41.285212 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/223f4ba1-8ccb-46d7-9606-59f8ee798577-webhook-cert\") pod \"223f4ba1-8ccb-46d7-9606-59f8ee798577\" (UID: \"223f4ba1-8ccb-46d7-9606-59f8ee798577\") " Jul 02 19:21:41.287439 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:41.287411 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223f4ba1-8ccb-46d7-9606-59f8ee798577-kube-api-access-w4c7p" (OuterVolumeSpecName: "kube-api-access-w4c7p") pod "223f4ba1-8ccb-46d7-9606-59f8ee798577" (UID: "223f4ba1-8ccb-46d7-9606-59f8ee798577"). InnerVolumeSpecName "kube-api-access-w4c7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:21:41.287632 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:41.287588 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/223f4ba1-8ccb-46d7-9606-59f8ee798577-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "223f4ba1-8ccb-46d7-9606-59f8ee798577" (UID: "223f4ba1-8ccb-46d7-9606-59f8ee798577"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:21:41.386066 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:41.385995 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4c7p\" (UniqueName: \"kubernetes.io/projected/223f4ba1-8ccb-46d7-9606-59f8ee798577-kube-api-access-w4c7p\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:21:41.386066 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:41.386021 2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/223f4ba1-8ccb-46d7-9606-59f8ee798577-webhook-cert\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:21:41.670034 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:41.669960 2562 generic.go:356] "Generic (PLEG): container finished" podID="223f4ba1-8ccb-46d7-9606-59f8ee798577" containerID="46be2dd4c0a43f023430ed5cdb0df1096e184ea2f78b73c22f1a78e5f58e7d89" exitCode=0 Jul 02 19:21:41.670034 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:41.670004 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8858bc75-q4wjb" Jul 02 19:21:41.670225 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:41.670042 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8858bc75-q4wjb" event={"ID":"223f4ba1-8ccb-46d7-9606-59f8ee798577","Type":"ContainerDied","Data":"46be2dd4c0a43f023430ed5cdb0df1096e184ea2f78b73c22f1a78e5f58e7d89"} Jul 02 19:21:41.670225 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:41.670093 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8858bc75-q4wjb" event={"ID":"223f4ba1-8ccb-46d7-9606-59f8ee798577","Type":"ContainerDied","Data":"74197fcc9c8f9bb7c9a57b05ea5e00eec837e7690a5203999139105ee4c9f5ed"} Jul 02 19:21:41.670225 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:41.670115 2562 scope.go:117] "RemoveContainer" containerID="46be2dd4c0a43f023430ed5cdb0df1096e184ea2f78b73c22f1a78e5f58e7d89" Jul 02 19:21:41.680013 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:41.679999 2562 scope.go:117] "RemoveContainer" containerID="46be2dd4c0a43f023430ed5cdb0df1096e184ea2f78b73c22f1a78e5f58e7d89" Jul 02 19:21:41.680261 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:21:41.680244 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46be2dd4c0a43f023430ed5cdb0df1096e184ea2f78b73c22f1a78e5f58e7d89\": container with ID starting with 46be2dd4c0a43f023430ed5cdb0df1096e184ea2f78b73c22f1a78e5f58e7d89 not found: ID does not exist" containerID="46be2dd4c0a43f023430ed5cdb0df1096e184ea2f78b73c22f1a78e5f58e7d89" Jul 02 19:21:41.680327 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:41.680267 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46be2dd4c0a43f023430ed5cdb0df1096e184ea2f78b73c22f1a78e5f58e7d89"} err="failed to get container status \"46be2dd4c0a43f023430ed5cdb0df1096e184ea2f78b73c22f1a78e5f58e7d89\": rpc error: code = NotFound desc = could not find container \"46be2dd4c0a43f023430ed5cdb0df1096e184ea2f78b73c22f1a78e5f58e7d89\": container with ID starting with 46be2dd4c0a43f023430ed5cdb0df1096e184ea2f78b73c22f1a78e5f58e7d89 not found: ID does not exist" Jul 02 19:21:41.692533 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:41.692511 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c8858bc75-q4wjb"] Jul 02 19:21:41.696358 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:41.696339 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c8858bc75-q4wjb"] Jul 02 19:21:42.358572 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:42.358541 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:21:42.475873 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:42.475840 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c8858bc75-t4lq9"] Jul 02 19:21:42.476319 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:42.476304 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="223f4ba1-8ccb-46d7-9606-59f8ee798577" containerName="manager" Jul 02 19:21:42.476377 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:42.476323 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="223f4ba1-8ccb-46d7-9606-59f8ee798577" containerName="manager" Jul 02 19:21:42.476414 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:42.476388 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="223f4ba1-8ccb-46d7-9606-59f8ee798577" containerName="manager" Jul 02 19:21:42.480519 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:42.480504 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8858bc75-t4lq9" Jul 02 19:21:42.484141 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:42.484123 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-9kkq8\"" Jul 02 19:21:42.484269 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:42.484149 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 02 19:21:42.484269 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:42.484123 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 02 19:21:42.488285 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:42.488262 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c8858bc75-t4lq9"] Jul 02 19:21:42.597173 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:42.597148 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e639f3e4-a42f-4252-9779-62c70a052603-webhook-cert\") pod \"maas-controller-c8858bc75-t4lq9\" (UID: \"e639f3e4-a42f-4252-9779-62c70a052603\") " pod="opendatahub/maas-controller-c8858bc75-t4lq9" Jul 02 19:21:42.597316 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:42.597190 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns7vc\" (UniqueName: \"kubernetes.io/projected/e639f3e4-a42f-4252-9779-62c70a052603-kube-api-access-ns7vc\") pod \"maas-controller-c8858bc75-t4lq9\" (UID: \"e639f3e4-a42f-4252-9779-62c70a052603\") " pod="opendatahub/maas-controller-c8858bc75-t4lq9" Jul 02 19:21:42.698568 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:42.698500 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e639f3e4-a42f-4252-9779-62c70a052603-webhook-cert\") pod \"maas-controller-c8858bc75-t4lq9\" (UID: \"e639f3e4-a42f-4252-9779-62c70a052603\") " pod="opendatahub/maas-controller-c8858bc75-t4lq9" Jul 02 19:21:42.698568 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:42.698542 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns7vc\" (UniqueName: \"kubernetes.io/projected/e639f3e4-a42f-4252-9779-62c70a052603-kube-api-access-ns7vc\") pod \"maas-controller-c8858bc75-t4lq9\" (UID: \"e639f3e4-a42f-4252-9779-62c70a052603\") " pod="opendatahub/maas-controller-c8858bc75-t4lq9" Jul 02 19:21:42.701127 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:42.701101 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e639f3e4-a42f-4252-9779-62c70a052603-webhook-cert\") pod \"maas-controller-c8858bc75-t4lq9\" (UID: \"e639f3e4-a42f-4252-9779-62c70a052603\") " pod="opendatahub/maas-controller-c8858bc75-t4lq9" Jul 02 19:21:42.707115 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:42.707096 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns7vc\" (UniqueName: \"kubernetes.io/projected/e639f3e4-a42f-4252-9779-62c70a052603-kube-api-access-ns7vc\") pod \"maas-controller-c8858bc75-t4lq9\" (UID: \"e639f3e4-a42f-4252-9779-62c70a052603\") " pod="opendatahub/maas-controller-c8858bc75-t4lq9" Jul 02 19:21:42.777272 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:42.777235 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="223f4ba1-8ccb-46d7-9606-59f8ee798577" path="/var/lib/kubelet/pods/223f4ba1-8ccb-46d7-9606-59f8ee798577/volumes" Jul 02 19:21:42.791259 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:42.791235 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c8858bc75-t4lq9" Jul 02 19:21:42.929460 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:42.929417 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c8858bc75-t4lq9"] Jul 02 19:21:42.932180 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:21:42.932155 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode639f3e4_a42f_4252_9779_62c70a052603.slice/crio-20a4ce4351e28d7e5f68e6136f1247359281a18504b356ff1d6dbec39012deaf WatchSource:0}: Error finding container 20a4ce4351e28d7e5f68e6136f1247359281a18504b356ff1d6dbec39012deaf: Status 404 returned error can't find the container with id 20a4ce4351e28d7e5f68e6136f1247359281a18504b356ff1d6dbec39012deaf Jul 02 19:21:43.681377 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:43.681340 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8858bc75-t4lq9" event={"ID":"e639f3e4-a42f-4252-9779-62c70a052603","Type":"ContainerStarted","Data":"20a4ce4351e28d7e5f68e6136f1247359281a18504b356ff1d6dbec39012deaf"} Jul 02 19:21:44.687751 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:44.687717 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c8858bc75-t4lq9" event={"ID":"e639f3e4-a42f-4252-9779-62c70a052603","Type":"ContainerStarted","Data":"400b4ef206679c74774194782337e2d5328b4d8dc3c59537dd2fe324d24c297d"} Jul 02 19:21:44.688140 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:44.687841 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c8858bc75-t4lq9" Jul 02 19:21:44.705231 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:44.705186 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c8858bc75-t4lq9" podStartSLOduration=1.934387753 podStartE2EDuration="2.705173461s" totalImagesPullingTime="770.785708ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:21:44.704177357 +0000 UTC m=+812.481194946" watchObservedRunningTime="2026-07-02 19:21:44.705173461 +0000 UTC m=+812.482191027" Jul 02 19:21:55.698029 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:21:55.697996 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c8858bc75-t4lq9" Jul 02 19:22:19.151429 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:22:19.151393 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:22:19.254078 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:22:19.254049 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:22:19.380037 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:22:19.380008 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:22:25.352548 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:22:25.352517 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:22:25.452804 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:22:25.452771 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:22:25.578797 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:22:25.578764 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:22:25.700423 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:22:25.700356 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:22:31.632441 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:22:31.632403 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:22:31.746985 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:22:31.746952 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:24:45.250932 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:24:45.250845 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:47.349881 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:27:47.349850 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:54.789726 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:27:54.789692 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:54.909347 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:27:54.909316 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:07.323182 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:28:07.323138 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:42.020416 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:28:42.020372 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:00.147907 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:30:00.147871 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29717010-427c6"] Jul 02 19:30:00.151410 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:30:00.151392 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717010-427c6" Jul 02 19:30:00.167227 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:30:00.167203 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29717010-427c6"] Jul 02 19:30:00.203024 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:30:00.202995 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcrkf\" (UniqueName: \"kubernetes.io/projected/6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903-kube-api-access-pcrkf\") pod \"maas-api-key-cleanup-29717010-427c6\" (UID: \"6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903\") " pod="opendatahub/maas-api-key-cleanup-29717010-427c6" Jul 02 19:30:00.304449 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:30:00.304418 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcrkf\" (UniqueName: \"kubernetes.io/projected/6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903-kube-api-access-pcrkf\") pod \"maas-api-key-cleanup-29717010-427c6\" (UID: \"6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903\") " pod="opendatahub/maas-api-key-cleanup-29717010-427c6" Jul 02 19:30:00.313770 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:30:00.313749 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcrkf\" (UniqueName: \"kubernetes.io/projected/6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903-kube-api-access-pcrkf\") pod \"maas-api-key-cleanup-29717010-427c6\" (UID: \"6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903\") " pod="opendatahub/maas-api-key-cleanup-29717010-427c6" Jul 02 19:30:00.461555 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:30:00.461478 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717010-427c6" Jul 02 19:30:00.606041 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:30:00.606012 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29717010-427c6"] Jul 02 19:30:00.607824 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:30:00.607799 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb3ae2d_f06c_4cf7_a01f_f19ba38b5903.slice/crio-3393ef79603899e606f81f4bb31872e412c811a0826c0b5ff6ba026f1dfe77cf WatchSource:0}: Error finding container 3393ef79603899e606f81f4bb31872e412c811a0826c0b5ff6ba026f1dfe77cf: Status 404 returned error can't find the container with id 3393ef79603899e606f81f4bb31872e412c811a0826c0b5ff6ba026f1dfe77cf Jul 02 19:30:00.609745 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:30:00.609729 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:30:01.150197 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:30:01.150167 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717010-427c6" event={"ID":"6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903","Type":"ContainerStarted","Data":"3393ef79603899e606f81f4bb31872e412c811a0826c0b5ff6ba026f1dfe77cf"} Jul 02 19:30:11.204254 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:30:11.204164 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717010-427c6" event={"ID":"6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903","Type":"ContainerStarted","Data":"d46d2ee5480849d51eb836973450a2ec034a5e2eec80a5733f1862c5a299f076"} Jul 02 19:30:11.222520 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:30:11.222464 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29717010-427c6" podStartSLOduration=1.353063298 podStartE2EDuration="11.222428661s" totalImagesPullingTime="9.869365363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:30:11.220786077 +0000 UTC m=+1318.997803669" watchObservedRunningTime="2026-07-02 19:30:11.222428661 +0000 UTC m=+1318.999446236" Jul 02 19:30:12.210657 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:30:12.210622 2562 generic.go:356] "Generic (PLEG): container finished" podID="6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903" containerID="d46d2ee5480849d51eb836973450a2ec034a5e2eec80a5733f1862c5a299f076" exitCode=0 Jul 02 19:30:12.211028 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:30:12.210685 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717010-427c6" event={"ID":"6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903","Type":"ContainerDied","Data":"d46d2ee5480849d51eb836973450a2ec034a5e2eec80a5733f1862c5a299f076"} Jul 02 19:30:13.355200 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:30:13.355177 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717010-427c6" Jul 02 19:30:13.425220 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:30:13.425192 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcrkf\" (UniqueName: \"kubernetes.io/projected/6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903-kube-api-access-pcrkf\") pod \"6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903\" (UID: \"6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903\") " Jul 02 19:30:13.427312 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:30:13.427285 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903-kube-api-access-pcrkf" (OuterVolumeSpecName: "kube-api-access-pcrkf") pod "6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903" (UID: "6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903"). InnerVolumeSpecName "kube-api-access-pcrkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:30:13.525976 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:30:13.525955 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcrkf\" (UniqueName: \"kubernetes.io/projected/6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903-kube-api-access-pcrkf\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:30:14.224320 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:30:14.224292 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717010-427c6" Jul 02 19:30:14.224506 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:30:14.224346 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717010-427c6" event={"ID":"6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903","Type":"ContainerDied","Data":"3393ef79603899e606f81f4bb31872e412c811a0826c0b5ff6ba026f1dfe77cf"} Jul 02 19:30:14.224506 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:30:14.224406 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3393ef79603899e606f81f4bb31872e412c811a0826c0b5ff6ba026f1dfe77cf" Jul 02 19:30:58.428223 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:30:58.428134 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:11.919127 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:11.919081 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:18.465330 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:18.465292 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:26.739336 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:26.739306 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:31.906866 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:31.906831 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:34.925124 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:34.925030 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:35.099146 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:35.099109 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:35.374110 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:35.374073 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:41.503402 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:41.503365 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:41.609298 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:41.609245 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:41.723425 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:41.723378 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:41.820428 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:41.820396 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:42.914984 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:42.914948 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:42.984901 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:42.984870 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr"] Jul 02 19:33:42.985302 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:42.985288 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903" containerName="cleanup" Jul 02 19:33:42.985302 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:42.985302 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903" containerName="cleanup" Jul 02 19:33:42.985412 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:42.985370 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903" containerName="cleanup" Jul 02 19:33:42.988379 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:42.988358 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr" Jul 02 19:33:42.991621 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:42.991600 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-3747bc9f-serving-cert\"" Jul 02 19:33:43.001336 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:43.001311 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr"] Jul 02 19:33:43.008239 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:43.008197 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:43.063887 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:43.063859 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca6d1b49-9eb5-4c68-8806-a47862a11506-maas-api-tls\") pod \"maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr\" (UID: \"ca6d1b49-9eb5-4c68-8806-a47862a11506\") " pod="opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr" Jul 02 19:33:43.064041 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:43.063985 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9d95\" (UniqueName: \"kubernetes.io/projected/ca6d1b49-9eb5-4c68-8806-a47862a11506-kube-api-access-n9d95\") pod \"maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr\" (UID: \"ca6d1b49-9eb5-4c68-8806-a47862a11506\") " pod="opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr" Jul 02 19:33:43.064101 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:43.064054 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca6d1b49-9eb5-4c68-8806-a47862a11506-openshift-service-ca\") pod \"maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr\" (UID: \"ca6d1b49-9eb5-4c68-8806-a47862a11506\") " pod="opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr" Jul 02 19:33:43.093043 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:43.093017 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:43.165245 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:43.165174 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca6d1b49-9eb5-4c68-8806-a47862a11506-maas-api-tls\") pod \"maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr\" (UID: \"ca6d1b49-9eb5-4c68-8806-a47862a11506\") " pod="opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr" Jul 02 19:33:43.165245 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:43.165232 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9d95\" (UniqueName: \"kubernetes.io/projected/ca6d1b49-9eb5-4c68-8806-a47862a11506-kube-api-access-n9d95\") pod \"maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr\" (UID: \"ca6d1b49-9eb5-4c68-8806-a47862a11506\") " pod="opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr" Jul 02 19:33:43.165405 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:43.165263 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca6d1b49-9eb5-4c68-8806-a47862a11506-openshift-service-ca\") pod \"maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr\" (UID: \"ca6d1b49-9eb5-4c68-8806-a47862a11506\") " pod="opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr" Jul 02 19:33:43.165985 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:43.165967 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca6d1b49-9eb5-4c68-8806-a47862a11506-openshift-service-ca\") pod \"maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr\" (UID: \"ca6d1b49-9eb5-4c68-8806-a47862a11506\") " pod="opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr" Jul 02 19:33:43.167963 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:43.167923 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca6d1b49-9eb5-4c68-8806-a47862a11506-maas-api-tls\") pod \"maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr\" (UID: \"ca6d1b49-9eb5-4c68-8806-a47862a11506\") " pod="opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr" Jul 02 19:33:43.177153 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:43.177126 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9d95\" (UniqueName: \"kubernetes.io/projected/ca6d1b49-9eb5-4c68-8806-a47862a11506-kube-api-access-n9d95\") pod \"maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr\" (UID: \"ca6d1b49-9eb5-4c68-8806-a47862a11506\") " pod="opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr" Jul 02 19:33:43.202910 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:43.202886 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:43.298760 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:43.298728 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr" Jul 02 19:33:43.441434 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:43.441412 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr"] Jul 02 19:33:43.443482 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:33:43.443452 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6d1b49_9eb5_4c68_8806_a47862a11506.slice/crio-29a32ab5674ff76bf7b8d8c10beec2eb76d90c54709f32d3d845ef3e9235066f WatchSource:0}: Error finding container 29a32ab5674ff76bf7b8d8c10beec2eb76d90c54709f32d3d845ef3e9235066f: Status 404 returned error can't find the container with id 29a32ab5674ff76bf7b8d8c10beec2eb76d90c54709f32d3d845ef3e9235066f Jul 02 19:33:43.549392 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:43.549356 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:43.714240 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:43.714148 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:44.275933 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:44.275901 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr" event={"ID":"ca6d1b49-9eb5-4c68-8806-a47862a11506","Type":"ContainerStarted","Data":"29a32ab5674ff76bf7b8d8c10beec2eb76d90c54709f32d3d845ef3e9235066f"} Jul 02 19:33:45.194814 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:45.194781 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:45.282527 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:45.282493 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr" event={"ID":"ca6d1b49-9eb5-4c68-8806-a47862a11506","Type":"ContainerStarted","Data":"6e991e5127721a8cb513789466148be9c2f4fc105369e024de44f8e565a9f225"} Jul 02 19:33:45.282938 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:45.282652 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr" Jul 02 19:33:45.301867 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:45.301825 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr" podStartSLOduration=2.499734117 podStartE2EDuration="3.301810788s" totalImagesPullingTime="802.076671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:33:45.30046227 +0000 UTC m=+1533.077479857" watchObservedRunningTime="2026-07-02 19:33:45.301810788 +0000 UTC m=+1533.078828354" Jul 02 19:33:48.033640 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:48.033560 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:48.211712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:48.211677 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:48.314923 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:48.314850 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:48.326833 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:48.326803 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr"] Jul 02 19:33:48.327036 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:48.327010 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr" podUID="ca6d1b49-9eb5-4c68-8806-a47862a11506" containerName="maas-api" containerID="cri-o://6e991e5127721a8cb513789466148be9c2f4fc105369e024de44f8e565a9f225" gracePeriod=30 Jul 02 19:33:48.381191 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:48.381151 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:48.571340 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:48.571284 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr" Jul 02 19:33:48.720652 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:48.720623 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca6d1b49-9eb5-4c68-8806-a47862a11506-maas-api-tls\") pod \"ca6d1b49-9eb5-4c68-8806-a47862a11506\" (UID: \"ca6d1b49-9eb5-4c68-8806-a47862a11506\") " Jul 02 19:33:48.720791 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:48.720684 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca6d1b49-9eb5-4c68-8806-a47862a11506-openshift-service-ca\") pod \"ca6d1b49-9eb5-4c68-8806-a47862a11506\" (UID: \"ca6d1b49-9eb5-4c68-8806-a47862a11506\") " Jul 02 19:33:48.720791 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:48.720758 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9d95\" (UniqueName: \"kubernetes.io/projected/ca6d1b49-9eb5-4c68-8806-a47862a11506-kube-api-access-n9d95\") pod \"ca6d1b49-9eb5-4c68-8806-a47862a11506\" (UID: \"ca6d1b49-9eb5-4c68-8806-a47862a11506\") " Jul 02 19:33:48.721049 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:48.721021 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca6d1b49-9eb5-4c68-8806-a47862a11506-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ca6d1b49-9eb5-4c68-8806-a47862a11506" (UID: "ca6d1b49-9eb5-4c68-8806-a47862a11506"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:33:48.722736 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:48.722710 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca6d1b49-9eb5-4c68-8806-a47862a11506-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ca6d1b49-9eb5-4c68-8806-a47862a11506" (UID: "ca6d1b49-9eb5-4c68-8806-a47862a11506"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:33:48.722846 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:48.722820 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca6d1b49-9eb5-4c68-8806-a47862a11506-kube-api-access-n9d95" (OuterVolumeSpecName: "kube-api-access-n9d95") pod "ca6d1b49-9eb5-4c68-8806-a47862a11506" (UID: "ca6d1b49-9eb5-4c68-8806-a47862a11506"). InnerVolumeSpecName "kube-api-access-n9d95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:33:48.821866 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:48.821794 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9d95\" (UniqueName: \"kubernetes.io/projected/ca6d1b49-9eb5-4c68-8806-a47862a11506-kube-api-access-n9d95\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:33:48.821866 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:48.821821 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca6d1b49-9eb5-4c68-8806-a47862a11506-maas-api-tls\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:33:48.821866 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:48.821832 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca6d1b49-9eb5-4c68-8806-a47862a11506-openshift-service-ca\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:33:49.304504 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:49.304471 2562 generic.go:356] "Generic (PLEG): container finished" podID="ca6d1b49-9eb5-4c68-8806-a47862a11506" containerID="6e991e5127721a8cb513789466148be9c2f4fc105369e024de44f8e565a9f225" exitCode=0 Jul 02 19:33:49.304929 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:49.304525 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr" Jul 02 19:33:49.304929 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:49.304575 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr" event={"ID":"ca6d1b49-9eb5-4c68-8806-a47862a11506","Type":"ContainerDied","Data":"6e991e5127721a8cb513789466148be9c2f4fc105369e024de44f8e565a9f225"} Jul 02 19:33:49.304929 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:49.304641 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr" event={"ID":"ca6d1b49-9eb5-4c68-8806-a47862a11506","Type":"ContainerDied","Data":"29a32ab5674ff76bf7b8d8c10beec2eb76d90c54709f32d3d845ef3e9235066f"} Jul 02 19:33:49.304929 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:49.304668 2562 scope.go:117] "RemoveContainer" containerID="6e991e5127721a8cb513789466148be9c2f4fc105369e024de44f8e565a9f225" Jul 02 19:33:49.314701 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:49.314686 2562 scope.go:117] "RemoveContainer" containerID="6e991e5127721a8cb513789466148be9c2f4fc105369e024de44f8e565a9f225" Jul 02 19:33:49.314991 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:33:49.314969 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e991e5127721a8cb513789466148be9c2f4fc105369e024de44f8e565a9f225\": container with ID starting with 6e991e5127721a8cb513789466148be9c2f4fc105369e024de44f8e565a9f225 not found: ID does not exist" containerID="6e991e5127721a8cb513789466148be9c2f4fc105369e024de44f8e565a9f225" Jul 02 19:33:49.315044 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:49.315005 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e991e5127721a8cb513789466148be9c2f4fc105369e024de44f8e565a9f225"} err="failed to get container status \"6e991e5127721a8cb513789466148be9c2f4fc105369e024de44f8e565a9f225\": rpc error: code = NotFound desc = could not find container \"6e991e5127721a8cb513789466148be9c2f4fc105369e024de44f8e565a9f225\": container with ID starting with 6e991e5127721a8cb513789466148be9c2f4fc105369e024de44f8e565a9f225 not found: ID does not exist" Jul 02 19:33:49.322075 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:49.322050 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr"] Jul 02 19:33:49.326107 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:49.326086 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-3747bc9f-7b794c78df-xq7gr"] Jul 02 19:33:50.063205 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:50.063172 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:50.777564 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:50.777518 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca6d1b49-9eb5-4c68-8806-a47862a11506" path="/var/lib/kubelet/pods/ca6d1b49-9eb5-4c68-8806-a47862a11506/volumes" Jul 02 19:33:52.195346 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:52.195312 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:54.317993 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:54.317958 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:54.388034 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:54.388001 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:54.541186 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:54.541155 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:54.607362 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:54.607267 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:55.794511 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:55.794466 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:55.899272 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:55.899239 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2"] Jul 02 19:33:55.899779 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:55.899761 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca6d1b49-9eb5-4c68-8806-a47862a11506" containerName="maas-api" Jul 02 19:33:55.899890 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:55.899781 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca6d1b49-9eb5-4c68-8806-a47862a11506" containerName="maas-api" Jul 02 19:33:55.899890 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:55.899871 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca6d1b49-9eb5-4c68-8806-a47862a11506" containerName="maas-api" Jul 02 19:33:55.904867 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:55.904848 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2" Jul 02 19:33:55.908043 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:55.908024 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-f765905c-serving-cert\"" Jul 02 19:33:55.923386 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:55.923362 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2"] Jul 02 19:33:55.983975 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:55.983943 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5347f9d3-e23c-4291-902b-3547b487192a-maas-api-tls\") pod \"maas-api-e2e-ait-f765905c-7687fcb877-s79q2\" (UID: \"5347f9d3-e23c-4291-902b-3547b487192a\") " pod="opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2" Jul 02 19:33:55.984136 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:55.983995 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5347f9d3-e23c-4291-902b-3547b487192a-openshift-service-ca\") pod \"maas-api-e2e-ait-f765905c-7687fcb877-s79q2\" (UID: \"5347f9d3-e23c-4291-902b-3547b487192a\") " pod="opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2" Jul 02 19:33:55.984189 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:55.984159 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf5tp\" (UniqueName: \"kubernetes.io/projected/5347f9d3-e23c-4291-902b-3547b487192a-kube-api-access-zf5tp\") pod \"maas-api-e2e-ait-f765905c-7687fcb877-s79q2\" (UID: \"5347f9d3-e23c-4291-902b-3547b487192a\") " pod="opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2" Jul 02 19:33:56.008458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:56.008414 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:56.085511 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:56.085444 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5347f9d3-e23c-4291-902b-3547b487192a-openshift-service-ca\") pod \"maas-api-e2e-ait-f765905c-7687fcb877-s79q2\" (UID: \"5347f9d3-e23c-4291-902b-3547b487192a\") " pod="opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2" Jul 02 19:33:56.085661 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:56.085529 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf5tp\" (UniqueName: \"kubernetes.io/projected/5347f9d3-e23c-4291-902b-3547b487192a-kube-api-access-zf5tp\") pod \"maas-api-e2e-ait-f765905c-7687fcb877-s79q2\" (UID: \"5347f9d3-e23c-4291-902b-3547b487192a\") " pod="opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2" Jul 02 19:33:56.085661 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:56.085571 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5347f9d3-e23c-4291-902b-3547b487192a-maas-api-tls\") pod \"maas-api-e2e-ait-f765905c-7687fcb877-s79q2\" (UID: \"5347f9d3-e23c-4291-902b-3547b487192a\") " pod="opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2" Jul 02 19:33:56.085746 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:33:56.085707 2562 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-f765905c-serving-cert: secret "maas-api-e2e-ait-f765905c-serving-cert" not found Jul 02 19:33:56.085785 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:33:56.085761 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5347f9d3-e23c-4291-902b-3547b487192a-maas-api-tls podName:5347f9d3-e23c-4291-902b-3547b487192a nodeName:}" failed. No retries permitted until 2026-07-02 19:33:56.58574401 +0000 UTC m=+1544.362761571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5347f9d3-e23c-4291-902b-3547b487192a-maas-api-tls") pod "maas-api-e2e-ait-f765905c-7687fcb877-s79q2" (UID: "5347f9d3-e23c-4291-902b-3547b487192a") : secret "maas-api-e2e-ait-f765905c-serving-cert" not found Jul 02 19:33:56.086179 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:56.086152 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5347f9d3-e23c-4291-902b-3547b487192a-openshift-service-ca\") pod \"maas-api-e2e-ait-f765905c-7687fcb877-s79q2\" (UID: \"5347f9d3-e23c-4291-902b-3547b487192a\") " pod="opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2" Jul 02 19:33:56.097322 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:56.097300 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf5tp\" (UniqueName: \"kubernetes.io/projected/5347f9d3-e23c-4291-902b-3547b487192a-kube-api-access-zf5tp\") pod \"maas-api-e2e-ait-f765905c-7687fcb877-s79q2\" (UID: \"5347f9d3-e23c-4291-902b-3547b487192a\") " pod="opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2" Jul 02 19:33:56.100129 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:56.100102 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:56.435991 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:56.435845 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:56.538686 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:56.538645 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:56.591747 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:56.591722 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5347f9d3-e23c-4291-902b-3547b487192a-maas-api-tls\") pod \"maas-api-e2e-ait-f765905c-7687fcb877-s79q2\" (UID: \"5347f9d3-e23c-4291-902b-3547b487192a\") " pod="opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2" Jul 02 19:33:56.594260 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:56.594230 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5347f9d3-e23c-4291-902b-3547b487192a-maas-api-tls\") pod \"maas-api-e2e-ait-f765905c-7687fcb877-s79q2\" (UID: \"5347f9d3-e23c-4291-902b-3547b487192a\") " pod="opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2" Jul 02 19:33:56.636119 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:56.636094 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:56.815736 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:56.815706 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2" Jul 02 19:33:56.964226 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:56.964204 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2"] Jul 02 19:33:56.966464 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:33:56.966433 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5347f9d3_e23c_4291_902b_3547b487192a.slice/crio-0802b760a6fcaeebc5d766e148c858bb789432c85d1d9325a20a9bfaa80089d2 WatchSource:0}: Error finding container 0802b760a6fcaeebc5d766e148c858bb789432c85d1d9325a20a9bfaa80089d2: Status 404 returned error can't find the container with id 0802b760a6fcaeebc5d766e148c858bb789432c85d1d9325a20a9bfaa80089d2 Jul 02 19:33:57.351609 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:57.351567 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2" event={"ID":"5347f9d3-e23c-4291-902b-3547b487192a","Type":"ContainerStarted","Data":"0802b760a6fcaeebc5d766e148c858bb789432c85d1d9325a20a9bfaa80089d2"} Jul 02 19:33:57.509581 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:57.509534 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:58.358097 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:58.358056 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2" event={"ID":"5347f9d3-e23c-4291-902b-3547b487192a","Type":"ContainerStarted","Data":"dcc93b26d08238939bb167e5e92628a6e6c03761e5856f0adf99d667aabb9dc2"} Jul 02 19:33:58.358560 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:58.358121 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2" Jul 02 19:33:58.376540 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:33:58.376488 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2" podStartSLOduration=2.724892134 podStartE2EDuration="3.37647318s" totalImagesPullingTime="651.581046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:33:58.374497475 +0000 UTC m=+1546.151515076" watchObservedRunningTime="2026-07-02 19:33:58.37647318 +0000 UTC m=+1546.153490747" Jul 02 19:34:00.865738 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:00.865701 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:01.334733 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:01.334700 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2"] Jul 02 19:34:01.334945 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:01.334920 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2" podUID="5347f9d3-e23c-4291-902b-3547b487192a" containerName="maas-api" containerID="cri-o://dcc93b26d08238939bb167e5e92628a6e6c03761e5856f0adf99d667aabb9dc2" gracePeriod=30 Jul 02 19:34:01.359377 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:01.359349 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:01.585136 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:01.585070 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2" Jul 02 19:34:01.636413 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:01.636378 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5347f9d3-e23c-4291-902b-3547b487192a-maas-api-tls\") pod \"5347f9d3-e23c-4291-902b-3547b487192a\" (UID: \"5347f9d3-e23c-4291-902b-3547b487192a\") " Jul 02 19:34:01.636558 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:01.636465 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5347f9d3-e23c-4291-902b-3547b487192a-openshift-service-ca\") pod \"5347f9d3-e23c-4291-902b-3547b487192a\" (UID: \"5347f9d3-e23c-4291-902b-3547b487192a\") " Jul 02 19:34:01.636558 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:01.636544 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zf5tp\" (UniqueName: \"kubernetes.io/projected/5347f9d3-e23c-4291-902b-3547b487192a-kube-api-access-zf5tp\") pod \"5347f9d3-e23c-4291-902b-3547b487192a\" (UID: \"5347f9d3-e23c-4291-902b-3547b487192a\") " Jul 02 19:34:01.636936 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:01.636899 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5347f9d3-e23c-4291-902b-3547b487192a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5347f9d3-e23c-4291-902b-3547b487192a" (UID: "5347f9d3-e23c-4291-902b-3547b487192a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:34:01.638534 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:01.638500 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5347f9d3-e23c-4291-902b-3547b487192a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5347f9d3-e23c-4291-902b-3547b487192a" (UID: "5347f9d3-e23c-4291-902b-3547b487192a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:34:01.638673 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:01.638618 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5347f9d3-e23c-4291-902b-3547b487192a-kube-api-access-zf5tp" (OuterVolumeSpecName: "kube-api-access-zf5tp") pod "5347f9d3-e23c-4291-902b-3547b487192a" (UID: "5347f9d3-e23c-4291-902b-3547b487192a"). InnerVolumeSpecName "kube-api-access-zf5tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:34:01.738246 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:01.738207 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf5tp\" (UniqueName: \"kubernetes.io/projected/5347f9d3-e23c-4291-902b-3547b487192a-kube-api-access-zf5tp\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:34:01.738246 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:01.738243 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5347f9d3-e23c-4291-902b-3547b487192a-maas-api-tls\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:34:01.738390 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:01.738257 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5347f9d3-e23c-4291-902b-3547b487192a-openshift-service-ca\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:34:02.381353 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:02.381321 2562 generic.go:356] "Generic (PLEG): container finished" podID="5347f9d3-e23c-4291-902b-3547b487192a" containerID="dcc93b26d08238939bb167e5e92628a6e6c03761e5856f0adf99d667aabb9dc2" exitCode=0 Jul 02 19:34:02.381799 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:02.381370 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2" Jul 02 19:34:02.381799 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:02.381414 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2" event={"ID":"5347f9d3-e23c-4291-902b-3547b487192a","Type":"ContainerDied","Data":"dcc93b26d08238939bb167e5e92628a6e6c03761e5856f0adf99d667aabb9dc2"} Jul 02 19:34:02.381799 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:02.381475 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2" event={"ID":"5347f9d3-e23c-4291-902b-3547b487192a","Type":"ContainerDied","Data":"0802b760a6fcaeebc5d766e148c858bb789432c85d1d9325a20a9bfaa80089d2"} Jul 02 19:34:02.381799 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:02.381511 2562 scope.go:117] "RemoveContainer" containerID="dcc93b26d08238939bb167e5e92628a6e6c03761e5856f0adf99d667aabb9dc2" Jul 02 19:34:02.391231 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:02.391217 2562 scope.go:117] "RemoveContainer" containerID="dcc93b26d08238939bb167e5e92628a6e6c03761e5856f0adf99d667aabb9dc2" Jul 02 19:34:02.391487 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:34:02.391469 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcc93b26d08238939bb167e5e92628a6e6c03761e5856f0adf99d667aabb9dc2\": container with ID starting with dcc93b26d08238939bb167e5e92628a6e6c03761e5856f0adf99d667aabb9dc2 not found: ID does not exist" containerID="dcc93b26d08238939bb167e5e92628a6e6c03761e5856f0adf99d667aabb9dc2" Jul 02 19:34:02.391548 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:02.391501 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcc93b26d08238939bb167e5e92628a6e6c03761e5856f0adf99d667aabb9dc2"} err="failed to get container status \"dcc93b26d08238939bb167e5e92628a6e6c03761e5856f0adf99d667aabb9dc2\": rpc error: code = NotFound desc = could not find container \"dcc93b26d08238939bb167e5e92628a6e6c03761e5856f0adf99d667aabb9dc2\": container with ID starting with dcc93b26d08238939bb167e5e92628a6e6c03761e5856f0adf99d667aabb9dc2 not found: ID does not exist" Jul 02 19:34:02.403185 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:02.403158 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2"] Jul 02 19:34:02.411395 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:02.411374 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-f765905c-7687fcb877-s79q2"] Jul 02 19:34:02.777338 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:02.777309 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5347f9d3-e23c-4291-902b-3547b487192a" path="/var/lib/kubelet/pods/5347f9d3-e23c-4291-902b-3547b487192a/volumes" Jul 02 19:34:07.230515 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:07.230428 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:07.333049 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:07.333019 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:08.686765 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:08.686733 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:13.065816 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:13.065728 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:13.200055 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:13.200016 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:13.390918 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:13.390889 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:13.484490 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:13.484454 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:14.538358 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:14.538176 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:14.607234 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:14.607198 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26"] Jul 02 19:34:14.607661 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:14.607647 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5347f9d3-e23c-4291-902b-3547b487192a" containerName="maas-api" Jul 02 19:34:14.607661 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:14.607661 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5347f9d3-e23c-4291-902b-3547b487192a" containerName="maas-api" Jul 02 19:34:14.607795 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:14.607734 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5347f9d3-e23c-4291-902b-3547b487192a" containerName="maas-api" Jul 02 19:34:14.611955 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:14.611933 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26" Jul 02 19:34:14.615566 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:14.615546 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-85690430-serving-cert\"" Jul 02 19:34:14.623475 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:14.623451 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26"] Jul 02 19:34:14.626022 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:14.625994 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:14.709247 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:14.709218 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:14.754734 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:14.754702 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5qzn\" (UniqueName: \"kubernetes.io/projected/1e00e090-a109-45f4-921b-e4083cdfb540-kube-api-access-b5qzn\") pod \"maas-api-e2e-derive-85690430-67f9cbb588-gkz26\" (UID: \"1e00e090-a109-45f4-921b-e4083cdfb540\") " pod="opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26" Jul 02 19:34:14.754902 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:14.754742 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e00e090-a109-45f4-921b-e4083cdfb540-openshift-service-ca\") pod \"maas-api-e2e-derive-85690430-67f9cbb588-gkz26\" (UID: \"1e00e090-a109-45f4-921b-e4083cdfb540\") " pod="opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26" Jul 02 19:34:14.754902 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:14.754783 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e00e090-a109-45f4-921b-e4083cdfb540-maas-api-tls\") pod \"maas-api-e2e-derive-85690430-67f9cbb588-gkz26\" (UID: \"1e00e090-a109-45f4-921b-e4083cdfb540\") " pod="opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26" Jul 02 19:34:14.856157 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:14.856069 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5qzn\" (UniqueName: \"kubernetes.io/projected/1e00e090-a109-45f4-921b-e4083cdfb540-kube-api-access-b5qzn\") pod \"maas-api-e2e-derive-85690430-67f9cbb588-gkz26\" (UID: \"1e00e090-a109-45f4-921b-e4083cdfb540\") " pod="opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26" Jul 02 19:34:14.856157 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:14.856118 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e00e090-a109-45f4-921b-e4083cdfb540-openshift-service-ca\") pod \"maas-api-e2e-derive-85690430-67f9cbb588-gkz26\" (UID: \"1e00e090-a109-45f4-921b-e4083cdfb540\") " pod="opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26" Jul 02 19:34:14.856374 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:14.856167 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e00e090-a109-45f4-921b-e4083cdfb540-maas-api-tls\") pod \"maas-api-e2e-derive-85690430-67f9cbb588-gkz26\" (UID: \"1e00e090-a109-45f4-921b-e4083cdfb540\") " pod="opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26" Jul 02 19:34:14.856374 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:34:14.856308 2562 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-85690430-serving-cert: secret "maas-api-e2e-derive-85690430-serving-cert" not found Jul 02 19:34:14.856490 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:34:14.856416 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e00e090-a109-45f4-921b-e4083cdfb540-maas-api-tls podName:1e00e090-a109-45f4-921b-e4083cdfb540 nodeName:}" failed. No retries permitted until 2026-07-02 19:34:15.356394276 +0000 UTC m=+1563.133411838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1e00e090-a109-45f4-921b-e4083cdfb540-maas-api-tls") pod "maas-api-e2e-derive-85690430-67f9cbb588-gkz26" (UID: "1e00e090-a109-45f4-921b-e4083cdfb540") : secret "maas-api-e2e-derive-85690430-serving-cert" not found Jul 02 19:34:14.856810 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:14.856788 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e00e090-a109-45f4-921b-e4083cdfb540-openshift-service-ca\") pod \"maas-api-e2e-derive-85690430-67f9cbb588-gkz26\" (UID: \"1e00e090-a109-45f4-921b-e4083cdfb540\") " pod="opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26" Jul 02 19:34:14.869415 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:14.869394 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5qzn\" (UniqueName: \"kubernetes.io/projected/1e00e090-a109-45f4-921b-e4083cdfb540-kube-api-access-b5qzn\") pod \"maas-api-e2e-derive-85690430-67f9cbb588-gkz26\" (UID: \"1e00e090-a109-45f4-921b-e4083cdfb540\") " pod="opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26" Jul 02 19:34:15.047813 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:15.047760 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:15.338338 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:15.338309 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:15.361105 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:15.361083 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e00e090-a109-45f4-921b-e4083cdfb540-maas-api-tls\") pod \"maas-api-e2e-derive-85690430-67f9cbb588-gkz26\" (UID: \"1e00e090-a109-45f4-921b-e4083cdfb540\") " pod="opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26" Jul 02 19:34:15.363749 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:15.363722 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e00e090-a109-45f4-921b-e4083cdfb540-maas-api-tls\") pod \"maas-api-e2e-derive-85690430-67f9cbb588-gkz26\" (UID: \"1e00e090-a109-45f4-921b-e4083cdfb540\") " pod="opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26" Jul 02 19:34:15.440473 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:15.440441 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:15.523881 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:15.523849 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26" Jul 02 19:34:15.674625 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:15.674587 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26"] Jul 02 19:34:15.676457 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:34:15.676431 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e00e090_a109_45f4_921b_e4083cdfb540.slice/crio-86cf7be320e7facf3176c8c5c4ddcb1fd2a42cc5ec08bd1984c200714819cdea WatchSource:0}: Error finding container 86cf7be320e7facf3176c8c5c4ddcb1fd2a42cc5ec08bd1984c200714819cdea: Status 404 returned error can't find the container with id 86cf7be320e7facf3176c8c5c4ddcb1fd2a42cc5ec08bd1984c200714819cdea Jul 02 19:34:16.451537 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:16.451451 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26" event={"ID":"1e00e090-a109-45f4-921b-e4083cdfb540","Type":"ContainerStarted","Data":"3af22a0d092c0201f33b5befb5fb93d13df481193c31542321bb3c59e4954fae"} Jul 02 19:34:16.451537 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:16.451507 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26" event={"ID":"1e00e090-a109-45f4-921b-e4083cdfb540","Type":"ContainerStarted","Data":"86cf7be320e7facf3176c8c5c4ddcb1fd2a42cc5ec08bd1984c200714819cdea"} Jul 02 19:34:16.451537 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:16.451535 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26" Jul 02 19:34:16.469157 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:16.469104 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26" podStartSLOduration=2.053398679 podStartE2EDuration="2.469087397s" totalImagesPullingTime="415.688718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:34:16.467655309 +0000 UTC m=+1564.244672897" watchObservedRunningTime="2026-07-02 19:34:16.469087397 +0000 UTC m=+1564.246104963" Jul 02 19:34:16.555027 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:16.554968 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:19.012022 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:19.011981 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:19.197935 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:19.197903 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:19.297733 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:19.297701 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:19.337089 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:19.337052 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26"] Jul 02 19:34:19.337425 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:19.337374 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26" podUID="1e00e090-a109-45f4-921b-e4083cdfb540" containerName="maas-api" containerID="cri-o://3af22a0d092c0201f33b5befb5fb93d13df481193c31542321bb3c59e4954fae" gracePeriod=30 Jul 02 19:34:19.367089 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:19.367058 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:19.471239 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:19.471213 2562 generic.go:356] "Generic (PLEG): container finished" podID="1e00e090-a109-45f4-921b-e4083cdfb540" containerID="3af22a0d092c0201f33b5befb5fb93d13df481193c31542321bb3c59e4954fae" exitCode=0 Jul 02 19:34:19.471370 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:19.471290 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26" event={"ID":"1e00e090-a109-45f4-921b-e4083cdfb540","Type":"ContainerDied","Data":"3af22a0d092c0201f33b5befb5fb93d13df481193c31542321bb3c59e4954fae"} Jul 02 19:34:19.587842 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:19.587816 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26" Jul 02 19:34:19.703529 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:19.703503 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e00e090-a109-45f4-921b-e4083cdfb540-openshift-service-ca\") pod \"1e00e090-a109-45f4-921b-e4083cdfb540\" (UID: \"1e00e090-a109-45f4-921b-e4083cdfb540\") " Jul 02 19:34:19.703833 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:19.703578 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5qzn\" (UniqueName: \"kubernetes.io/projected/1e00e090-a109-45f4-921b-e4083cdfb540-kube-api-access-b5qzn\") pod \"1e00e090-a109-45f4-921b-e4083cdfb540\" (UID: \"1e00e090-a109-45f4-921b-e4083cdfb540\") " Jul 02 19:34:19.703833 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:19.703674 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e00e090-a109-45f4-921b-e4083cdfb540-maas-api-tls\") pod \"1e00e090-a109-45f4-921b-e4083cdfb540\" (UID: \"1e00e090-a109-45f4-921b-e4083cdfb540\") " Jul 02 19:34:19.703833 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:19.703821 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e00e090-a109-45f4-921b-e4083cdfb540-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1e00e090-a109-45f4-921b-e4083cdfb540" (UID: "1e00e090-a109-45f4-921b-e4083cdfb540"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:34:19.704071 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:19.704049 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e00e090-a109-45f4-921b-e4083cdfb540-openshift-service-ca\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:34:19.705718 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:19.705697 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e00e090-a109-45f4-921b-e4083cdfb540-kube-api-access-b5qzn" (OuterVolumeSpecName: "kube-api-access-b5qzn") pod "1e00e090-a109-45f4-921b-e4083cdfb540" (UID: "1e00e090-a109-45f4-921b-e4083cdfb540"). InnerVolumeSpecName "kube-api-access-b5qzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:34:19.705956 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:19.705937 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e00e090-a109-45f4-921b-e4083cdfb540-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1e00e090-a109-45f4-921b-e4083cdfb540" (UID: "1e00e090-a109-45f4-921b-e4083cdfb540"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:34:19.805136 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:19.805105 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e00e090-a109-45f4-921b-e4083cdfb540-maas-api-tls\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:34:19.805136 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:19.805133 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5qzn\" (UniqueName: \"kubernetes.io/projected/1e00e090-a109-45f4-921b-e4083cdfb540-kube-api-access-b5qzn\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:34:20.477507 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:20.477473 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26" event={"ID":"1e00e090-a109-45f4-921b-e4083cdfb540","Type":"ContainerDied","Data":"86cf7be320e7facf3176c8c5c4ddcb1fd2a42cc5ec08bd1984c200714819cdea"} Jul 02 19:34:20.477507 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:20.477504 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26" Jul 02 19:34:20.477989 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:20.477556 2562 scope.go:117] "RemoveContainer" containerID="3af22a0d092c0201f33b5befb5fb93d13df481193c31542321bb3c59e4954fae" Jul 02 19:34:20.498702 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:20.498677 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26"] Jul 02 19:34:20.505137 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:20.505114 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-85690430-67f9cbb588-gkz26"] Jul 02 19:34:20.777707 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:20.777674 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e00e090-a109-45f4-921b-e4083cdfb540" path="/var/lib/kubelet/pods/1e00e090-a109-45f4-921b-e4083cdfb540/volumes" Jul 02 19:34:21.090986 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:21.090956 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:25.049398 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:25.049361 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:25.111835 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:25.111809 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:25.570940 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:25.570885 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:25.673908 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:25.673881 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:25.767410 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:25.767364 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:27.229536 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:27.229465 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:33.045833 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:33.045792 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:33.116817 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:33.116782 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:33.197643 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:33.197578 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:33.296175 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:33.296092 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:33.396103 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:33.396055 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:33.792223 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:33.792192 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:59.906003 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:34:59.905966 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:06.093236 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:35:06.093149 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:06.200754 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:35:06.200666 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:06.382839 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:35:06.382758 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:22.253013 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:35:22.252929 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:28.192459 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:35:28.192426 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:28.296840 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:35:28.296802 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:28.398967 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:35:28.398932 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:38.943255 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:35:38.943221 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:39.794929 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:35:39.794894 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:46.113173 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:35:46.113135 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:46.196673 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:35:46.196642 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:46.319552 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:35:46.319506 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:46.413453 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:35:46.413383 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:47.275114 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:35:47.275084 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:47.368168 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:35:47.368128 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:49.283510 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:35:49.283477 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:55.560517 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:35:55.560472 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:55.701283 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:35:55.701253 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:56.365813 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:35:56.365779 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:02.287237 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:02.287199 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:04.131264 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:04.131229 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:06.480445 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:06.480397 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:06.553358 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:06.553324 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr"] Jul 02 19:36:06.553784 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:06.553770 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e00e090-a109-45f4-921b-e4083cdfb540" containerName="maas-api" Jul 02 19:36:06.553784 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:06.553785 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e00e090-a109-45f4-921b-e4083cdfb540" containerName="maas-api" Jul 02 19:36:06.553943 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:06.553931 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e00e090-a109-45f4-921b-e4083cdfb540" containerName="maas-api" Jul 02 19:36:06.557029 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:06.557013 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr" Jul 02 19:36:06.560130 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:06.560109 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-2cd582b7-serving-cert\"" Jul 02 19:36:06.570324 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:06.570299 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr"] Jul 02 19:36:06.592744 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:06.592723 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:06.661155 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:06.661130 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16ef75a2-872c-4565-a80b-57977021d542-maas-api-tls\") pod \"maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr\" (UID: \"16ef75a2-872c-4565-a80b-57977021d542\") " pod="opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr" Jul 02 19:36:06.661271 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:06.661163 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ef75a2-872c-4565-a80b-57977021d542-openshift-service-ca\") pod \"maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr\" (UID: \"16ef75a2-872c-4565-a80b-57977021d542\") " pod="opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr" Jul 02 19:36:06.661271 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:06.661199 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcqqc\" (UniqueName: \"kubernetes.io/projected/16ef75a2-872c-4565-a80b-57977021d542-kube-api-access-jcqqc\") pod \"maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr\" (UID: \"16ef75a2-872c-4565-a80b-57977021d542\") " pod="opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr" Jul 02 19:36:06.697165 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:06.697132 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:06.761851 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:06.761772 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16ef75a2-872c-4565-a80b-57977021d542-maas-api-tls\") pod \"maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr\" (UID: \"16ef75a2-872c-4565-a80b-57977021d542\") " pod="opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr" Jul 02 19:36:06.761851 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:06.761813 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ef75a2-872c-4565-a80b-57977021d542-openshift-service-ca\") pod \"maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr\" (UID: \"16ef75a2-872c-4565-a80b-57977021d542\") " pod="opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr" Jul 02 19:36:06.761851 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:06.761842 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcqqc\" (UniqueName: \"kubernetes.io/projected/16ef75a2-872c-4565-a80b-57977021d542-kube-api-access-jcqqc\") pod \"maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr\" (UID: \"16ef75a2-872c-4565-a80b-57977021d542\") " pod="opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr" Jul 02 19:36:06.762468 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:06.762447 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ef75a2-872c-4565-a80b-57977021d542-openshift-service-ca\") pod \"maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr\" (UID: \"16ef75a2-872c-4565-a80b-57977021d542\") " pod="opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr" Jul 02 19:36:06.764386 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:06.764368 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16ef75a2-872c-4565-a80b-57977021d542-maas-api-tls\") pod \"maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr\" (UID: \"16ef75a2-872c-4565-a80b-57977021d542\") " pod="opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr" Jul 02 19:36:06.770507 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:06.770484 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcqqc\" (UniqueName: \"kubernetes.io/projected/16ef75a2-872c-4565-a80b-57977021d542-kube-api-access-jcqqc\") pod \"maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr\" (UID: \"16ef75a2-872c-4565-a80b-57977021d542\") " pod="opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr" Jul 02 19:36:06.785011 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:06.784987 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:06.867748 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:06.867711 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr" Jul 02 19:36:07.045128 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:07.045106 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr"] Jul 02 19:36:07.047401 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:36:07.047376 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ef75a2_872c_4565_a80b_57977021d542.slice/crio-18e45e4b908ea1d32cf45b3513a16ed8710ca4970df52d03eb964ac6d7ef822a WatchSource:0}: Error finding container 18e45e4b908ea1d32cf45b3513a16ed8710ca4970df52d03eb964ac6d7ef822a: Status 404 returned error can't find the container with id 18e45e4b908ea1d32cf45b3513a16ed8710ca4970df52d03eb964ac6d7ef822a Jul 02 19:36:07.049063 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:07.049042 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:36:07.108622 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:07.108571 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:07.204984 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:07.204953 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:08.016116 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:08.016088 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr" event={"ID":"16ef75a2-872c-4565-a80b-57977021d542","Type":"ContainerStarted","Data":"74acda37d3f18f8dd929e1ddd8c6a9b0c238efd6e00b004523033062bac5997b"} Jul 02 19:36:08.016473 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:08.016133 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr" event={"ID":"16ef75a2-872c-4565-a80b-57977021d542","Type":"ContainerStarted","Data":"18e45e4b908ea1d32cf45b3513a16ed8710ca4970df52d03eb964ac6d7ef822a"} Jul 02 19:36:08.016473 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:08.016224 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr" Jul 02 19:36:08.043613 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:08.043538 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr" podStartSLOduration=1.6479198990000001 podStartE2EDuration="2.043519273s" totalImagesPullingTime="395.599374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:36:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:36:08.042194423 +0000 UTC m=+1675.819211986" watchObservedRunningTime="2026-07-02 19:36:08.043519273 +0000 UTC m=+1675.820536840" Jul 02 19:36:10.987346 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:10.987308 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:11.685695 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:11.685650 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:11.781669 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:11.781633 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:12.141780 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:12.141744 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:14.027665 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:14.027642 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr" Jul 02 19:36:18.020854 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:18.020823 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:18.118695 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:18.118664 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:18.442142 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:18.442065 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:18.752388 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:18.752308 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr"] Jul 02 19:36:18.752858 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:18.752799 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr" podUID="16ef75a2-872c-4565-a80b-57977021d542" containerName="maas-api" containerID="cri-o://74acda37d3f18f8dd929e1ddd8c6a9b0c238efd6e00b004523033062bac5997b" gracePeriod=30 Jul 02 19:36:18.807563 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:18.807526 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:18.902305 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:18.902272 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:18.992502 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:18.992479 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr" Jul 02 19:36:19.073682 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:19.073657 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ef75a2-872c-4565-a80b-57977021d542-openshift-service-ca\") pod \"16ef75a2-872c-4565-a80b-57977021d542\" (UID: \"16ef75a2-872c-4565-a80b-57977021d542\") " Jul 02 19:36:19.074131 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:19.073760 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcqqc\" (UniqueName: \"kubernetes.io/projected/16ef75a2-872c-4565-a80b-57977021d542-kube-api-access-jcqqc\") pod \"16ef75a2-872c-4565-a80b-57977021d542\" (UID: \"16ef75a2-872c-4565-a80b-57977021d542\") " Jul 02 19:36:19.074131 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:19.073841 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16ef75a2-872c-4565-a80b-57977021d542-maas-api-tls\") pod \"16ef75a2-872c-4565-a80b-57977021d542\" (UID: \"16ef75a2-872c-4565-a80b-57977021d542\") " Jul 02 19:36:19.074131 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:19.074023 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ef75a2-872c-4565-a80b-57977021d542-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "16ef75a2-872c-4565-a80b-57977021d542" (UID: "16ef75a2-872c-4565-a80b-57977021d542"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:36:19.074302 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:19.074193 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ef75a2-872c-4565-a80b-57977021d542-openshift-service-ca\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:36:19.074945 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:19.074919 2562 generic.go:356] "Generic (PLEG): container finished" podID="16ef75a2-872c-4565-a80b-57977021d542" containerID="74acda37d3f18f8dd929e1ddd8c6a9b0c238efd6e00b004523033062bac5997b" exitCode=0 Jul 02 19:36:19.075067 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:19.074966 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr" Jul 02 19:36:19.075067 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:19.074993 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr" event={"ID":"16ef75a2-872c-4565-a80b-57977021d542","Type":"ContainerDied","Data":"74acda37d3f18f8dd929e1ddd8c6a9b0c238efd6e00b004523033062bac5997b"} Jul 02 19:36:19.075067 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:19.075061 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr" event={"ID":"16ef75a2-872c-4565-a80b-57977021d542","Type":"ContainerDied","Data":"18e45e4b908ea1d32cf45b3513a16ed8710ca4970df52d03eb964ac6d7ef822a"} Jul 02 19:36:19.075178 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:19.075095 2562 scope.go:117] "RemoveContainer" containerID="74acda37d3f18f8dd929e1ddd8c6a9b0c238efd6e00b004523033062bac5997b" Jul 02 19:36:19.076305 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:19.076145 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16ef75a2-872c-4565-a80b-57977021d542-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "16ef75a2-872c-4565-a80b-57977021d542" (UID: "16ef75a2-872c-4565-a80b-57977021d542"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:36:19.076305 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:19.076224 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ef75a2-872c-4565-a80b-57977021d542-kube-api-access-jcqqc" (OuterVolumeSpecName: "kube-api-access-jcqqc") pod "16ef75a2-872c-4565-a80b-57977021d542" (UID: "16ef75a2-872c-4565-a80b-57977021d542"). InnerVolumeSpecName "kube-api-access-jcqqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:36:19.095277 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:19.095260 2562 scope.go:117] "RemoveContainer" containerID="74acda37d3f18f8dd929e1ddd8c6a9b0c238efd6e00b004523033062bac5997b" Jul 02 19:36:19.095518 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:36:19.095502 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74acda37d3f18f8dd929e1ddd8c6a9b0c238efd6e00b004523033062bac5997b\": container with ID starting with 74acda37d3f18f8dd929e1ddd8c6a9b0c238efd6e00b004523033062bac5997b not found: ID does not exist" containerID="74acda37d3f18f8dd929e1ddd8c6a9b0c238efd6e00b004523033062bac5997b" Jul 02 19:36:19.095566 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:19.095526 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74acda37d3f18f8dd929e1ddd8c6a9b0c238efd6e00b004523033062bac5997b"} err="failed to get container status \"74acda37d3f18f8dd929e1ddd8c6a9b0c238efd6e00b004523033062bac5997b\": rpc error: code = NotFound desc = could not find container \"74acda37d3f18f8dd929e1ddd8c6a9b0c238efd6e00b004523033062bac5997b\": container with ID starting with 74acda37d3f18f8dd929e1ddd8c6a9b0c238efd6e00b004523033062bac5997b not found: ID does not exist" Jul 02 19:36:19.175154 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:19.175134 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcqqc\" (UniqueName: \"kubernetes.io/projected/16ef75a2-872c-4565-a80b-57977021d542-kube-api-access-jcqqc\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:36:19.175154 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:19.175155 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16ef75a2-872c-4565-a80b-57977021d542-maas-api-tls\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:36:19.398083 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:19.398052 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr"] Jul 02 19:36:19.403305 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:19.403275 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-2cd582b7-59cd94c9b6-5hktr"] Jul 02 19:36:20.777217 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:20.777188 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16ef75a2-872c-4565-a80b-57977021d542" path="/var/lib/kubelet/pods/16ef75a2-872c-4565-a80b-57977021d542/volumes" Jul 02 19:36:25.019382 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:25.019340 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:25.091546 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:25.091512 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:25.450962 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:25.450879 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:25.590557 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:25.590522 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:25.738971 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:25.738887 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:25.838283 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:25.838254 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:25.939378 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:25.939346 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:31.375177 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:31.375133 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:31.472123 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:31.472086 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:32.079114 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:32.079072 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:32.194553 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:32.194524 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:32.298216 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:32.298179 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:32.439944 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:32.439850 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:38.157352 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:38.157318 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:38.225283 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:38.225242 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:44.720630 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:44.720570 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:45.133694 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:45.133644 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:45.247405 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:45.247353 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:48.680822 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:48.680785 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:48.732942 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:48.732910 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc"] Jul 02 19:36:48.733361 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:48.733349 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16ef75a2-872c-4565-a80b-57977021d542" containerName="maas-api" Jul 02 19:36:48.733404 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:48.733363 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="16ef75a2-872c-4565-a80b-57977021d542" containerName="maas-api" Jul 02 19:36:48.733453 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:48.733445 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="16ef75a2-872c-4565-a80b-57977021d542" containerName="maas-api" Jul 02 19:36:48.736849 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:48.736833 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" Jul 02 19:36:48.739456 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:48.739429 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-a603bec7-serving-cert\"" Jul 02 19:36:48.748225 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:48.748198 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc"] Jul 02 19:36:48.748697 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:48.748666 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/049752dc-d61b-45b3-896b-f465063b1839-openshift-service-ca\") pod \"maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc\" (UID: \"049752dc-d61b-45b3-896b-f465063b1839\") " pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" Jul 02 19:36:48.748849 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:48.748831 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/049752dc-d61b-45b3-896b-f465063b1839-maas-api-tls\") pod \"maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc\" (UID: \"049752dc-d61b-45b3-896b-f465063b1839\") " pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" Jul 02 19:36:48.748956 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:48.748935 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cd25\" (UniqueName: \"kubernetes.io/projected/049752dc-d61b-45b3-896b-f465063b1839-kube-api-access-6cd25\") pod \"maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc\" (UID: \"049752dc-d61b-45b3-896b-f465063b1839\") " pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" Jul 02 19:36:48.758830 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:48.758801 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:48.836871 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:48.836835 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:48.849997 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:48.849968 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/049752dc-d61b-45b3-896b-f465063b1839-maas-api-tls\") pod \"maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc\" (UID: \"049752dc-d61b-45b3-896b-f465063b1839\") " pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" Jul 02 19:36:48.850157 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:48.850022 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cd25\" (UniqueName: \"kubernetes.io/projected/049752dc-d61b-45b3-896b-f465063b1839-kube-api-access-6cd25\") pod \"maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc\" (UID: \"049752dc-d61b-45b3-896b-f465063b1839\") " pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" Jul 02 19:36:48.850157 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:48.850086 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/049752dc-d61b-45b3-896b-f465063b1839-openshift-service-ca\") pod \"maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc\" (UID: \"049752dc-d61b-45b3-896b-f465063b1839\") " pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" Jul 02 19:36:48.850300 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:36:48.850183 2562 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-a603bec7-serving-cert: secret "maas-api-e2e-mt-a603bec7-serving-cert" not found Jul 02 19:36:48.850357 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:36:48.850304 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/049752dc-d61b-45b3-896b-f465063b1839-maas-api-tls podName:049752dc-d61b-45b3-896b-f465063b1839 nodeName:}" failed. No retries permitted until 2026-07-02 19:36:49.350271785 +0000 UTC m=+1717.127289347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/049752dc-d61b-45b3-896b-f465063b1839-maas-api-tls") pod "maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" (UID: "049752dc-d61b-45b3-896b-f465063b1839") : secret "maas-api-e2e-mt-a603bec7-serving-cert" not found Jul 02 19:36:48.850790 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:48.850770 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/049752dc-d61b-45b3-896b-f465063b1839-openshift-service-ca\") pod \"maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc\" (UID: \"049752dc-d61b-45b3-896b-f465063b1839\") " pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" Jul 02 19:36:48.862695 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:48.862667 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cd25\" (UniqueName: \"kubernetes.io/projected/049752dc-d61b-45b3-896b-f465063b1839-kube-api-access-6cd25\") pod \"maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc\" (UID: \"049752dc-d61b-45b3-896b-f465063b1839\") " pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" Jul 02 19:36:48.937365 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:48.937284 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:49.231487 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:49.231394 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:49.330905 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:49.330871 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:49.356212 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:49.356185 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/049752dc-d61b-45b3-896b-f465063b1839-maas-api-tls\") pod \"maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc\" (UID: \"049752dc-d61b-45b3-896b-f465063b1839\") " pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" Jul 02 19:36:49.358712 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:49.358689 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/049752dc-d61b-45b3-896b-f465063b1839-maas-api-tls\") pod \"maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc\" (UID: \"049752dc-d61b-45b3-896b-f465063b1839\") " pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" Jul 02 19:36:49.649621 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:49.649578 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" Jul 02 19:36:49.789576 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:49.789549 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc"] Jul 02 19:36:49.791681 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:36:49.791649 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049752dc_d61b_45b3_896b_f465063b1839.slice/crio-d99393670fb7a0d182b03c5c20dcbdab84531c13c1c8087bd02b34f821419913 WatchSource:0}: Error finding container d99393670fb7a0d182b03c5c20dcbdab84531c13c1c8087bd02b34f821419913: Status 404 returned error can't find the container with id d99393670fb7a0d182b03c5c20dcbdab84531c13c1c8087bd02b34f821419913 Jul 02 19:36:50.243771 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:50.243740 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" event={"ID":"049752dc-d61b-45b3-896b-f465063b1839","Type":"ContainerStarted","Data":"b89b3e7e6b20d678b465df297b694eaa34e2b50eac78795801af8836af154709"} Jul 02 19:36:50.243874 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:50.243784 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" event={"ID":"049752dc-d61b-45b3-896b-f465063b1839","Type":"ContainerStarted","Data":"d99393670fb7a0d182b03c5c20dcbdab84531c13c1c8087bd02b34f821419913"} Jul 02 19:36:50.243874 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:50.243860 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" Jul 02 19:36:50.263494 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:50.263450 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" podStartSLOduration=1.8819446420000001 podStartE2EDuration="2.263437218s" totalImagesPullingTime="381.492576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:36:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:36:50.261516975 +0000 UTC m=+1718.038534564" watchObservedRunningTime="2026-07-02 19:36:50.263437218 +0000 UTC m=+1718.040454814" Jul 02 19:36:53.203422 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:53.203391 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:56.254775 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:36:56.254748 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" Jul 02 19:37:05.023158 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:05.023125 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:37:05.123662 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:05.123625 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:37:10.014475 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:10.014401 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:37:10.115202 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:10.115165 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:37:13.121489 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:13.121384 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:37:30.879441 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:30.879397 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:37:31.184416 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.184316 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc"] Jul 02 19:37:31.184671 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.184642 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" podUID="049752dc-d61b-45b3-896b-f465063b1839" containerName="maas-api" containerID="cri-o://b89b3e7e6b20d678b465df297b694eaa34e2b50eac78795801af8836af154709" gracePeriod=30 Jul 02 19:37:31.209962 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.209927 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:37:31.250116 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.250081 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" podUID="049752dc-d61b-45b3-896b-f465063b1839" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.45:8443/health\": dial tcp 10.134.0.45:8443: connect: connection refused" Jul 02 19:37:31.441304 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.441252 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" Jul 02 19:37:31.458024 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.457989 2562 generic.go:356] "Generic (PLEG): container finished" podID="049752dc-d61b-45b3-896b-f465063b1839" containerID="b89b3e7e6b20d678b465df297b694eaa34e2b50eac78795801af8836af154709" exitCode=0 Jul 02 19:37:31.458140 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.458031 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" Jul 02 19:37:31.458140 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.458071 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" event={"ID":"049752dc-d61b-45b3-896b-f465063b1839","Type":"ContainerDied","Data":"b89b3e7e6b20d678b465df297b694eaa34e2b50eac78795801af8836af154709"} Jul 02 19:37:31.458140 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.458127 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc" event={"ID":"049752dc-d61b-45b3-896b-f465063b1839","Type":"ContainerDied","Data":"d99393670fb7a0d182b03c5c20dcbdab84531c13c1c8087bd02b34f821419913"} Jul 02 19:37:31.458288 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.458150 2562 scope.go:117] "RemoveContainer" containerID="b89b3e7e6b20d678b465df297b694eaa34e2b50eac78795801af8836af154709" Jul 02 19:37:31.464113 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.463540 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cd25\" (UniqueName: \"kubernetes.io/projected/049752dc-d61b-45b3-896b-f465063b1839-kube-api-access-6cd25\") pod \"049752dc-d61b-45b3-896b-f465063b1839\" (UID: \"049752dc-d61b-45b3-896b-f465063b1839\") " Jul 02 19:37:31.464113 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.463693 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/049752dc-d61b-45b3-896b-f465063b1839-maas-api-tls\") pod \"049752dc-d61b-45b3-896b-f465063b1839\" (UID: \"049752dc-d61b-45b3-896b-f465063b1839\") " Jul 02 19:37:31.464113 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.463752 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/049752dc-d61b-45b3-896b-f465063b1839-openshift-service-ca\") pod \"049752dc-d61b-45b3-896b-f465063b1839\" (UID: \"049752dc-d61b-45b3-896b-f465063b1839\") " Jul 02 19:37:31.464715 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.464582 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049752dc-d61b-45b3-896b-f465063b1839-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "049752dc-d61b-45b3-896b-f465063b1839" (UID: "049752dc-d61b-45b3-896b-f465063b1839"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:37:31.466477 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.466450 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/049752dc-d61b-45b3-896b-f465063b1839-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "049752dc-d61b-45b3-896b-f465063b1839" (UID: "049752dc-d61b-45b3-896b-f465063b1839"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:37:31.466585 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.466571 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049752dc-d61b-45b3-896b-f465063b1839-kube-api-access-6cd25" (OuterVolumeSpecName: "kube-api-access-6cd25") pod "049752dc-d61b-45b3-896b-f465063b1839" (UID: "049752dc-d61b-45b3-896b-f465063b1839"). InnerVolumeSpecName "kube-api-access-6cd25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:37:31.470292 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.470272 2562 scope.go:117] "RemoveContainer" containerID="b89b3e7e6b20d678b465df297b694eaa34e2b50eac78795801af8836af154709" Jul 02 19:37:31.470628 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:37:31.470586 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b89b3e7e6b20d678b465df297b694eaa34e2b50eac78795801af8836af154709\": container with ID starting with b89b3e7e6b20d678b465df297b694eaa34e2b50eac78795801af8836af154709 not found: ID does not exist" containerID="b89b3e7e6b20d678b465df297b694eaa34e2b50eac78795801af8836af154709" Jul 02 19:37:31.470716 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.470638 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b89b3e7e6b20d678b465df297b694eaa34e2b50eac78795801af8836af154709"} err="failed to get container status \"b89b3e7e6b20d678b465df297b694eaa34e2b50eac78795801af8836af154709\": rpc error: code = NotFound desc = could not find container \"b89b3e7e6b20d678b465df297b694eaa34e2b50eac78795801af8836af154709\": container with ID starting with b89b3e7e6b20d678b465df297b694eaa34e2b50eac78795801af8836af154709 not found: ID does not exist" Jul 02 19:37:31.565050 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.565023 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cd25\" (UniqueName: \"kubernetes.io/projected/049752dc-d61b-45b3-896b-f465063b1839-kube-api-access-6cd25\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:37:31.565050 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.565045 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/049752dc-d61b-45b3-896b-f465063b1839-maas-api-tls\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:37:31.565050 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.565055 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/049752dc-d61b-45b3-896b-f465063b1839-openshift-service-ca\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:37:31.779995 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.779964 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc"] Jul 02 19:37:31.784192 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:31.784165 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-a603bec7-6bb46dc594-ff5sc"] Jul 02 19:37:32.777883 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:32.777850 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="049752dc-d61b-45b3-896b-f465063b1839" path="/var/lib/kubelet/pods/049752dc-d61b-45b3-896b-f465063b1839/volumes" Jul 02 19:37:34.731687 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:34.731655 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:37:37.117005 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:37.116973 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:37:42.032361 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:37:42.032327 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:38:18.131769 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:38:18.131726 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:38:19.028338 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:38:19.028299 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:38:24.746944 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:38:24.746905 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:38:24.848732 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:38:24.848696 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:38:25.051044 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:38:25.051013 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:38:25.315364 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:38:25.315285 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:38:26.519680 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:38:26.519640 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:38:26.621323 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:38:26.621287 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:38:27.499699 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:38:27.499666 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:38:27.589097 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:38:27.589065 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:38:27.690345 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:38:27.690301 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:38:29.149048 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:38:29.149017 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:38:35.066722 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:38:35.066688 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:38:35.130521 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:38:35.130492 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:38:35.781338 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:38:35.781287 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:38:41.623069 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:38:41.623036 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:38:41.680094 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:38:41.680067 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:38:41.759376 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:38:41.759339 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:38:41.858483 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:38:41.858455 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:38:52.359349 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:38:52.359307 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:38:52.719089 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:38:52.718997 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:13.716661 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:13.716617 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:19.680356 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:19.680317 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:19.780484 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:19.780444 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:20.045920 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:20.045884 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:20.357297 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:20.357203 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:20.572285 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:20.572240 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:23.542003 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:23.541943 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:23.619465 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:23.619415 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4"] Jul 02 19:39:23.620095 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:23.620068 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="049752dc-d61b-45b3-896b-f465063b1839" containerName="maas-api" Jul 02 19:39:23.620095 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:23.620091 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="049752dc-d61b-45b3-896b-f465063b1839" containerName="maas-api" Jul 02 19:39:23.620326 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:23.620229 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="049752dc-d61b-45b3-896b-f465063b1839" containerName="maas-api" Jul 02 19:39:23.623488 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:23.623470 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" Jul 02 19:39:23.626874 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:23.626842 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-09f1f6-serving-cert\"" Jul 02 19:39:23.636308 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:23.636285 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4"] Jul 02 19:39:23.639724 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:23.639698 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:23.703195 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:23.703168 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144-openshift-service-ca\") pod \"maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4\" (UID: \"ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144\") " pod="opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" Jul 02 19:39:23.703315 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:23.703216 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144-maas-api-tls\") pod \"maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4\" (UID: \"ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144\") " pod="opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" Jul 02 19:39:23.703315 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:23.703248 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crjpg\" (UniqueName: \"kubernetes.io/projected/ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144-kube-api-access-crjpg\") pod \"maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4\" (UID: \"ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144\") " pod="opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" Jul 02 19:39:23.725484 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:23.725452 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:23.804714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:23.804636 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144-maas-api-tls\") pod \"maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4\" (UID: \"ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144\") " pod="opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" Jul 02 19:39:23.804714 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:23.804681 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crjpg\" (UniqueName: \"kubernetes.io/projected/ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144-kube-api-access-crjpg\") pod \"maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4\" (UID: \"ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144\") " pod="opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" Jul 02 19:39:23.804899 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:39:23.804794 2562 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-09f1f6-serving-cert: secret "maas-api-e2e-inf-a-09f1f6-serving-cert" not found Jul 02 19:39:23.804899 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:23.804823 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144-openshift-service-ca\") pod \"maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4\" (UID: \"ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144\") " pod="opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" Jul 02 19:39:23.804899 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:39:23.804865 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144-maas-api-tls podName:ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144 nodeName:}" failed. No retries permitted until 2026-07-02 19:39:24.304845429 +0000 UTC m=+1872.081862975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144-maas-api-tls") pod "maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" (UID: "ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144") : secret "maas-api-e2e-inf-a-09f1f6-serving-cert" not found Jul 02 19:39:23.805417 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:23.805398 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144-openshift-service-ca\") pod \"maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4\" (UID: \"ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144\") " pod="opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" Jul 02 19:39:23.815399 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:23.815374 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crjpg\" (UniqueName: \"kubernetes.io/projected/ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144-kube-api-access-crjpg\") pod \"maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4\" (UID: \"ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144\") " pod="opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" Jul 02 19:39:23.898443 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:23.898409 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:24.226373 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:24.226280 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:24.310111 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:24.310076 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144-maas-api-tls\") pod \"maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4\" (UID: \"ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144\") " pod="opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" Jul 02 19:39:24.312705 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:24.312678 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144-maas-api-tls\") pod \"maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4\" (UID: \"ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144\") " pod="opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" Jul 02 19:39:24.327048 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:24.327020 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:24.535466 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:24.535433 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" Jul 02 19:39:24.677286 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:24.677262 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4"] Jul 02 19:39:24.679525 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:39:24.679500 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5a3b7f_85d2_43a7_bfdb_5f53bf5cf144.slice/crio-b3bf57af191b44b2a267f98286ce6992d2b718a3405b64cac16fa8375f1b039c WatchSource:0}: Error finding container b3bf57af191b44b2a267f98286ce6992d2b718a3405b64cac16fa8375f1b039c: Status 404 returned error can't find the container with id b3bf57af191b44b2a267f98286ce6992d2b718a3405b64cac16fa8375f1b039c Jul 02 19:39:25.025203 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:25.025171 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" event={"ID":"ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144","Type":"ContainerStarted","Data":"b3bf57af191b44b2a267f98286ce6992d2b718a3405b64cac16fa8375f1b039c"} Jul 02 19:39:26.031354 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:26.031315 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" event={"ID":"ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144","Type":"ContainerStarted","Data":"e3b6edd16fa29e12f2cab1f32108fa6721250c0b94f1ddc81c6dde7a7c84d4dd"} Jul 02 19:39:26.031744 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:26.031376 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" Jul 02 19:39:26.050254 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:26.050202 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" podStartSLOduration=2.624637491 podStartE2EDuration="3.050178509s" totalImagesPullingTime="425.541018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:39:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:39:26.047784586 +0000 UTC m=+1873.824802174" watchObservedRunningTime="2026-07-02 19:39:26.050178509 +0000 UTC m=+1873.827196075" Jul 02 19:39:28.065412 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:28.065365 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:28.423369 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:28.423277 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:28.737439 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:28.737353 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:28.895410 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:28.895366 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:31.588667 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:31.588629 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:31.634129 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:31.634099 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p"] Jul 02 19:39:31.638883 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:31.638856 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" Jul 02 19:39:31.648655 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:31.647806 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-cc922a-serving-cert\"" Jul 02 19:39:31.652847 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:31.652820 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p"] Jul 02 19:39:31.678843 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:31.678809 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:31.682856 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:31.682836 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/554c3c46-ce74-4e06-83e4-4973ad122aab-maas-api-tls\") pod \"maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p\" (UID: \"554c3c46-ce74-4e06-83e4-4973ad122aab\") " pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" Jul 02 19:39:31.682952 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:31.682907 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88dbs\" (UniqueName: \"kubernetes.io/projected/554c3c46-ce74-4e06-83e4-4973ad122aab-kube-api-access-88dbs\") pod \"maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p\" (UID: \"554c3c46-ce74-4e06-83e4-4973ad122aab\") " pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" Jul 02 19:39:31.683011 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:31.682962 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/554c3c46-ce74-4e06-83e4-4973ad122aab-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p\" (UID: \"554c3c46-ce74-4e06-83e4-4973ad122aab\") " pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" Jul 02 19:39:31.752082 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:31.752047 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:31.783744 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:31.783717 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88dbs\" (UniqueName: \"kubernetes.io/projected/554c3c46-ce74-4e06-83e4-4973ad122aab-kube-api-access-88dbs\") pod \"maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p\" (UID: \"554c3c46-ce74-4e06-83e4-4973ad122aab\") " pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" Jul 02 19:39:31.783887 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:31.783759 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/554c3c46-ce74-4e06-83e4-4973ad122aab-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p\" (UID: \"554c3c46-ce74-4e06-83e4-4973ad122aab\") " pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" Jul 02 19:39:31.783887 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:31.783805 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/554c3c46-ce74-4e06-83e4-4973ad122aab-maas-api-tls\") pod \"maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p\" (UID: \"554c3c46-ce74-4e06-83e4-4973ad122aab\") " pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" Jul 02 19:39:31.783980 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:39:31.783913 2562 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-cc922a-serving-cert: secret "maas-api-e2e-inf-b-cc922a-serving-cert" not found Jul 02 19:39:31.784022 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:39:31.783986 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/554c3c46-ce74-4e06-83e4-4973ad122aab-maas-api-tls podName:554c3c46-ce74-4e06-83e4-4973ad122aab nodeName:}" failed. No retries permitted until 2026-07-02 19:39:32.283964546 +0000 UTC m=+1880.060982094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/554c3c46-ce74-4e06-83e4-4973ad122aab-maas-api-tls") pod "maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" (UID: "554c3c46-ce74-4e06-83e4-4973ad122aab") : secret "maas-api-e2e-inf-b-cc922a-serving-cert" not found Jul 02 19:39:31.784459 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:31.784440 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/554c3c46-ce74-4e06-83e4-4973ad122aab-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p\" (UID: \"554c3c46-ce74-4e06-83e4-4973ad122aab\") " pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" Jul 02 19:39:31.794774 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:31.794749 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88dbs\" (UniqueName: \"kubernetes.io/projected/554c3c46-ce74-4e06-83e4-4973ad122aab-kube-api-access-88dbs\") pod \"maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p\" (UID: \"554c3c46-ce74-4e06-83e4-4973ad122aab\") " pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" Jul 02 19:39:32.044857 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:32.044830 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" Jul 02 19:39:32.106006 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:32.105973 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:32.204802 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:32.204770 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:32.288609 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:32.288573 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/554c3c46-ce74-4e06-83e4-4973ad122aab-maas-api-tls\") pod \"maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p\" (UID: \"554c3c46-ce74-4e06-83e4-4973ad122aab\") " pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" Jul 02 19:39:32.290986 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:32.290959 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/554c3c46-ce74-4e06-83e4-4973ad122aab-maas-api-tls\") pod \"maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p\" (UID: \"554c3c46-ce74-4e06-83e4-4973ad122aab\") " pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" Jul 02 19:39:32.306174 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:32.306092 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:32.562025 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:32.561945 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" Jul 02 19:39:32.703990 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:32.703967 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p"] Jul 02 19:39:32.706024 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:39:32.705984 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod554c3c46_ce74_4e06_83e4_4973ad122aab.slice/crio-041b49bd14113d31bae99debfe14afe968dd60bb94bf114c6f9f1918509520c3 WatchSource:0}: Error finding container 041b49bd14113d31bae99debfe14afe968dd60bb94bf114c6f9f1918509520c3: Status 404 returned error can't find the container with id 041b49bd14113d31bae99debfe14afe968dd60bb94bf114c6f9f1918509520c3 Jul 02 19:39:33.068478 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:33.068450 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" event={"ID":"554c3c46-ce74-4e06-83e4-4973ad122aab","Type":"ContainerStarted","Data":"041b49bd14113d31bae99debfe14afe968dd60bb94bf114c6f9f1918509520c3"} Jul 02 19:39:34.074287 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:34.074253 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" event={"ID":"554c3c46-ce74-4e06-83e4-4973ad122aab","Type":"ContainerStarted","Data":"64f9973476fceb63b88e65abea2051bc71a2407b766569d85528130361e3daee"} Jul 02 19:39:34.074677 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:34.074314 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" Jul 02 19:39:34.094550 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:34.094497 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" podStartSLOduration=2.736936548 podStartE2EDuration="3.094481748s" totalImagesPullingTime="357.5452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:39:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:39:34.091118444 +0000 UTC m=+1881.868136028" watchObservedRunningTime="2026-07-02 19:39:34.094481748 +0000 UTC m=+1881.871499333" Jul 02 19:39:36.096696 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:36.096664 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:40.085481 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:40.085442 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" Jul 02 19:39:44.932997 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:44.932954 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:45.031715 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:45.031686 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:52.960350 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:52.960314 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:53.050497 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:39:53.050468 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:40:00.087695 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:00.087647 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:40:28.234464 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:28.234428 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:40:50.014484 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:50.014450 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:40:51.136115 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:51.136075 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:40:51.384325 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:51.384289 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:40:51.777880 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:51.777844 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:40:51.892884 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:51.892835 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:40:52.085845 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.085756 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4"] Jul 02 19:40:52.086054 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.086019 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" podUID="ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144" containerName="maas-api" containerID="cri-o://e3b6edd16fa29e12f2cab1f32108fa6721250c0b94f1ddc81c6dde7a7c84d4dd" gracePeriod=30 Jul 02 19:40:52.108846 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.108741 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:40:52.334880 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.334854 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" Jul 02 19:40:52.439348 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.439273 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144-maas-api-tls\") pod \"ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144\" (UID: \"ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144\") " Jul 02 19:40:52.439348 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.439315 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144-openshift-service-ca\") pod \"ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144\" (UID: \"ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144\") " Jul 02 19:40:52.439531 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.439357 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crjpg\" (UniqueName: \"kubernetes.io/projected/ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144-kube-api-access-crjpg\") pod \"ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144\" (UID: \"ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144\") " Jul 02 19:40:52.439733 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.439708 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144" (UID: "ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:40:52.441552 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.441528 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144" (UID: "ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:40:52.441682 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.441618 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144-kube-api-access-crjpg" (OuterVolumeSpecName: "kube-api-access-crjpg") pod "ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144" (UID: "ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144"). InnerVolumeSpecName "kube-api-access-crjpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:40:52.483326 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.483300 2562 generic.go:356] "Generic (PLEG): container finished" podID="ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144" containerID="e3b6edd16fa29e12f2cab1f32108fa6721250c0b94f1ddc81c6dde7a7c84d4dd" exitCode=0 Jul 02 19:40:52.483427 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.483349 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" Jul 02 19:40:52.483427 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.483389 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" event={"ID":"ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144","Type":"ContainerDied","Data":"e3b6edd16fa29e12f2cab1f32108fa6721250c0b94f1ddc81c6dde7a7c84d4dd"} Jul 02 19:40:52.483506 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.483445 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4" event={"ID":"ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144","Type":"ContainerDied","Data":"b3bf57af191b44b2a267f98286ce6992d2b718a3405b64cac16fa8375f1b039c"} Jul 02 19:40:52.483506 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.483475 2562 scope.go:117] "RemoveContainer" containerID="e3b6edd16fa29e12f2cab1f32108fa6721250c0b94f1ddc81c6dde7a7c84d4dd" Jul 02 19:40:52.492900 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.492883 2562 scope.go:117] "RemoveContainer" containerID="e3b6edd16fa29e12f2cab1f32108fa6721250c0b94f1ddc81c6dde7a7c84d4dd" Jul 02 19:40:52.493164 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:40:52.493144 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3b6edd16fa29e12f2cab1f32108fa6721250c0b94f1ddc81c6dde7a7c84d4dd\": container with ID starting with e3b6edd16fa29e12f2cab1f32108fa6721250c0b94f1ddc81c6dde7a7c84d4dd not found: ID does not exist" containerID="e3b6edd16fa29e12f2cab1f32108fa6721250c0b94f1ddc81c6dde7a7c84d4dd" Jul 02 19:40:52.493214 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.493171 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3b6edd16fa29e12f2cab1f32108fa6721250c0b94f1ddc81c6dde7a7c84d4dd"} err="failed to get container status \"e3b6edd16fa29e12f2cab1f32108fa6721250c0b94f1ddc81c6dde7a7c84d4dd\": rpc error: code = NotFound desc = could not find container \"e3b6edd16fa29e12f2cab1f32108fa6721250c0b94f1ddc81c6dde7a7c84d4dd\": container with ID starting with e3b6edd16fa29e12f2cab1f32108fa6721250c0b94f1ddc81c6dde7a7c84d4dd not found: ID does not exist" Jul 02 19:40:52.506564 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.506447 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4"] Jul 02 19:40:52.508556 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.508533 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-09f1f6-85d57b87f7-9vdp4"] Jul 02 19:40:52.540753 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.540732 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144-maas-api-tls\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:40:52.540845 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.540755 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144-openshift-service-ca\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:40:52.540845 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.540768 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crjpg\" (UniqueName: \"kubernetes.io/projected/ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144-kube-api-access-crjpg\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:40:52.778554 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:52.778525 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144" path="/var/lib/kubelet/pods/ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144/volumes" Jul 02 19:40:56.810818 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:56.810782 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:40:57.896957 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:40:57.896921 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:33.717237 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:33.717198 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:35.021843 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.021798 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p"] Jul 02 19:41:35.022302 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.022068 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" podUID="554c3c46-ce74-4e06-83e4-4973ad122aab" containerName="maas-api" containerID="cri-o://64f9973476fceb63b88e65abea2051bc71a2407b766569d85528130361e3daee" gracePeriod=30 Jul 02 19:41:35.080804 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.080768 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" podUID="554c3c46-ce74-4e06-83e4-4973ad122aab" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.47:8443/health\": dial tcp 10.134.0.47:8443: connect: connection refused" Jul 02 19:41:35.272620 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.272531 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" Jul 02 19:41:35.341661 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.341635 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88dbs\" (UniqueName: \"kubernetes.io/projected/554c3c46-ce74-4e06-83e4-4973ad122aab-kube-api-access-88dbs\") pod \"554c3c46-ce74-4e06-83e4-4973ad122aab\" (UID: \"554c3c46-ce74-4e06-83e4-4973ad122aab\") " Jul 02 19:41:35.341797 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.341730 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/554c3c46-ce74-4e06-83e4-4973ad122aab-openshift-service-ca\") pod \"554c3c46-ce74-4e06-83e4-4973ad122aab\" (UID: \"554c3c46-ce74-4e06-83e4-4973ad122aab\") " Jul 02 19:41:35.341843 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.341830 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/554c3c46-ce74-4e06-83e4-4973ad122aab-maas-api-tls\") pod \"554c3c46-ce74-4e06-83e4-4973ad122aab\" (UID: \"554c3c46-ce74-4e06-83e4-4973ad122aab\") " Jul 02 19:41:35.342135 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.342101 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/554c3c46-ce74-4e06-83e4-4973ad122aab-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "554c3c46-ce74-4e06-83e4-4973ad122aab" (UID: "554c3c46-ce74-4e06-83e4-4973ad122aab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:41:35.343918 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.343891 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/554c3c46-ce74-4e06-83e4-4973ad122aab-kube-api-access-88dbs" (OuterVolumeSpecName: "kube-api-access-88dbs") pod "554c3c46-ce74-4e06-83e4-4973ad122aab" (UID: "554c3c46-ce74-4e06-83e4-4973ad122aab"). InnerVolumeSpecName "kube-api-access-88dbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:41:35.344001 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.343954 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/554c3c46-ce74-4e06-83e4-4973ad122aab-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "554c3c46-ce74-4e06-83e4-4973ad122aab" (UID: "554c3c46-ce74-4e06-83e4-4973ad122aab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:41:35.443245 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.443216 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/554c3c46-ce74-4e06-83e4-4973ad122aab-maas-api-tls\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:41:35.443245 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.443241 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88dbs\" (UniqueName: \"kubernetes.io/projected/554c3c46-ce74-4e06-83e4-4973ad122aab-kube-api-access-88dbs\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:41:35.443245 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.443250 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/554c3c46-ce74-4e06-83e4-4973ad122aab-openshift-service-ca\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:41:35.456159 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.456132 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:35.710105 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.710026 2562 generic.go:356] "Generic (PLEG): container finished" podID="554c3c46-ce74-4e06-83e4-4973ad122aab" containerID="64f9973476fceb63b88e65abea2051bc71a2407b766569d85528130361e3daee" exitCode=0 Jul 02 19:41:35.710105 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.710074 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" Jul 02 19:41:35.710105 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.710076 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" event={"ID":"554c3c46-ce74-4e06-83e4-4973ad122aab","Type":"ContainerDied","Data":"64f9973476fceb63b88e65abea2051bc71a2407b766569d85528130361e3daee"} Jul 02 19:41:35.710314 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.710131 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p" event={"ID":"554c3c46-ce74-4e06-83e4-4973ad122aab","Type":"ContainerDied","Data":"041b49bd14113d31bae99debfe14afe968dd60bb94bf114c6f9f1918509520c3"} Jul 02 19:41:35.710314 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.710158 2562 scope.go:117] "RemoveContainer" containerID="64f9973476fceb63b88e65abea2051bc71a2407b766569d85528130361e3daee" Jul 02 19:41:35.719934 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.719832 2562 scope.go:117] "RemoveContainer" containerID="64f9973476fceb63b88e65abea2051bc71a2407b766569d85528130361e3daee" Jul 02 19:41:35.720094 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:41:35.720076 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f9973476fceb63b88e65abea2051bc71a2407b766569d85528130361e3daee\": container with ID starting with 64f9973476fceb63b88e65abea2051bc71a2407b766569d85528130361e3daee not found: ID does not exist" containerID="64f9973476fceb63b88e65abea2051bc71a2407b766569d85528130361e3daee" Jul 02 19:41:35.720140 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.720103 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f9973476fceb63b88e65abea2051bc71a2407b766569d85528130361e3daee"} err="failed to get container status \"64f9973476fceb63b88e65abea2051bc71a2407b766569d85528130361e3daee\": rpc error: code = NotFound desc = could not find container \"64f9973476fceb63b88e65abea2051bc71a2407b766569d85528130361e3daee\": container with ID starting with 64f9973476fceb63b88e65abea2051bc71a2407b766569d85528130361e3daee not found: ID does not exist" Jul 02 19:41:35.733023 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.733001 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p"] Jul 02 19:41:35.734876 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:35.734858 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-cc922a-5bf66466b-8x57p"] Jul 02 19:41:36.777265 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:36.777233 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="554c3c46-ce74-4e06-83e4-4973ad122aab" path="/var/lib/kubelet/pods/554c3c46-ce74-4e06-83e4-4973ad122aab/volumes" Jul 02 19:41:40.771633 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:40.771569 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:40.870837 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:40.870808 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:41.067865 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:41.067777 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:41.363069 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:41.362978 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:41.703757 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:41.703675 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:42.051985 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:42.051944 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:42.498663 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:42.498560 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:45.542449 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:45.542413 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:45.630011 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:45.629977 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:45.762412 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:45.762383 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c"] Jul 02 19:41:45.762839 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:45.762825 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144" containerName="maas-api" Jul 02 19:41:45.762889 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:45.762842 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144" containerName="maas-api" Jul 02 19:41:45.762889 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:45.762862 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="554c3c46-ce74-4e06-83e4-4973ad122aab" containerName="maas-api" Jul 02 19:41:45.762889 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:45.762867 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="554c3c46-ce74-4e06-83e4-4973ad122aab" containerName="maas-api" Jul 02 19:41:45.762985 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:45.762954 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee5a3b7f-85d2-43a7-bfdb-5f53bf5cf144" containerName="maas-api" Jul 02 19:41:45.762985 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:45.762962 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="554c3c46-ce74-4e06-83e4-4973ad122aab" containerName="maas-api" Jul 02 19:41:45.766108 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:45.766092 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c" Jul 02 19:41:45.769913 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:45.769886 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-a91b31-serving-cert\"" Jul 02 19:41:45.780217 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:45.780195 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c"] Jul 02 19:41:45.812029 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:45.811921 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:45.833101 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:45.833071 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/573b76fd-5bb0-4658-8426-ef85cd44d383-openshift-service-ca\") pod \"maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c\" (UID: \"573b76fd-5bb0-4658-8426-ef85cd44d383\") " pod="opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c" Jul 02 19:41:45.833218 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:45.833114 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/573b76fd-5bb0-4658-8426-ef85cd44d383-maas-api-tls\") pod \"maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c\" (UID: \"573b76fd-5bb0-4658-8426-ef85cd44d383\") " pod="opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c" Jul 02 19:41:45.833218 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:45.833196 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfws8\" (UniqueName: \"kubernetes.io/projected/573b76fd-5bb0-4658-8426-ef85cd44d383-kube-api-access-jfws8\") pod \"maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c\" (UID: \"573b76fd-5bb0-4658-8426-ef85cd44d383\") " pod="opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c" Jul 02 19:41:45.934469 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:45.934437 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/573b76fd-5bb0-4658-8426-ef85cd44d383-openshift-service-ca\") pod \"maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c\" (UID: \"573b76fd-5bb0-4658-8426-ef85cd44d383\") " pod="opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c" Jul 02 19:41:45.934650 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:45.934476 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/573b76fd-5bb0-4658-8426-ef85cd44d383-maas-api-tls\") pod \"maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c\" (UID: \"573b76fd-5bb0-4658-8426-ef85cd44d383\") " pod="opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c" Jul 02 19:41:45.934650 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:45.934522 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfws8\" (UniqueName: \"kubernetes.io/projected/573b76fd-5bb0-4658-8426-ef85cd44d383-kube-api-access-jfws8\") pod \"maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c\" (UID: \"573b76fd-5bb0-4658-8426-ef85cd44d383\") " pod="opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c" Jul 02 19:41:45.935206 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:45.935183 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/573b76fd-5bb0-4658-8426-ef85cd44d383-openshift-service-ca\") pod \"maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c\" (UID: \"573b76fd-5bb0-4658-8426-ef85cd44d383\") " pod="opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c" Jul 02 19:41:45.937233 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:45.937212 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/573b76fd-5bb0-4658-8426-ef85cd44d383-maas-api-tls\") pod \"maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c\" (UID: \"573b76fd-5bb0-4658-8426-ef85cd44d383\") " pod="opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c" Jul 02 19:41:45.958279 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:45.958254 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfws8\" (UniqueName: \"kubernetes.io/projected/573b76fd-5bb0-4658-8426-ef85cd44d383-kube-api-access-jfws8\") pod \"maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c\" (UID: \"573b76fd-5bb0-4658-8426-ef85cd44d383\") " pod="opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c" Jul 02 19:41:46.028133 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:46.028104 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:46.076357 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:46.076294 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c" Jul 02 19:41:46.212660 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:46.212635 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c"] Jul 02 19:41:46.214836 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:41:46.214808 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod573b76fd_5bb0_4658_8426_ef85cd44d383.slice/crio-893de9abf510436174ced0ead1b98f512c75775adeedf053f49b0546ea4e5cef WatchSource:0}: Error finding container 893de9abf510436174ced0ead1b98f512c75775adeedf053f49b0546ea4e5cef: Status 404 returned error can't find the container with id 893de9abf510436174ced0ead1b98f512c75775adeedf053f49b0546ea4e5cef Jul 02 19:41:46.216218 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:46.216202 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:41:46.395837 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:46.395748 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:46.498419 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:46.498385 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:46.766717 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:46.766621 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c" event={"ID":"573b76fd-5bb0-4658-8426-ef85cd44d383","Type":"ContainerStarted","Data":"ce33e19034ce696398a77705ab6b0cef623d01afd4d614f09d6fc9aad961e107"} Jul 02 19:41:46.766717 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:46.766679 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c" event={"ID":"573b76fd-5bb0-4658-8426-ef85cd44d383","Type":"ContainerStarted","Data":"893de9abf510436174ced0ead1b98f512c75775adeedf053f49b0546ea4e5cef"} Jul 02 19:41:46.767096 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:46.766832 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c" Jul 02 19:41:46.785176 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:46.785129 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c" podStartSLOduration=1.460865314 podStartE2EDuration="1.785114106s" totalImagesPullingTime="324.248792ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:41:46.783891484 +0000 UTC m=+2014.560909053" watchObservedRunningTime="2026-07-02 19:41:46.785114106 +0000 UTC m=+2014.562131673" Jul 02 19:41:50.095075 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:50.095038 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:50.529283 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:50.529244 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:50.879233 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:50.879147 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:51.042711 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:51.042673 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:52.777534 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:52.777503 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c" Jul 02 19:41:54.449573 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:54.449533 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:54.507028 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:54.506995 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f"] Jul 02 19:41:54.510490 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:54.510472 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" Jul 02 19:41:54.513839 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:54.513814 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-5a2a19-serving-cert\"" Jul 02 19:41:54.523541 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:54.523517 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f"] Jul 02 19:41:54.551517 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:54.551479 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:54.611405 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:54.611373 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06530fb0-aeed-4c86-84dd-4c66edc1bbce-maas-api-tls\") pod \"maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f\" (UID: \"06530fb0-aeed-4c86-84dd-4c66edc1bbce\") " pod="opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" Jul 02 19:41:54.611551 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:54.611409 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck479\" (UniqueName: \"kubernetes.io/projected/06530fb0-aeed-4c86-84dd-4c66edc1bbce-kube-api-access-ck479\") pod \"maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f\" (UID: \"06530fb0-aeed-4c86-84dd-4c66edc1bbce\") " pod="opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" Jul 02 19:41:54.611551 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:54.611502 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06530fb0-aeed-4c86-84dd-4c66edc1bbce-openshift-service-ca\") pod \"maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f\" (UID: \"06530fb0-aeed-4c86-84dd-4c66edc1bbce\") " pod="opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" Jul 02 19:41:54.644327 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:54.644299 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:54.712885 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:54.712815 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06530fb0-aeed-4c86-84dd-4c66edc1bbce-maas-api-tls\") pod \"maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f\" (UID: \"06530fb0-aeed-4c86-84dd-4c66edc1bbce\") " pod="opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" Jul 02 19:41:54.712885 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:54.712853 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck479\" (UniqueName: \"kubernetes.io/projected/06530fb0-aeed-4c86-84dd-4c66edc1bbce-kube-api-access-ck479\") pod \"maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f\" (UID: \"06530fb0-aeed-4c86-84dd-4c66edc1bbce\") " pod="opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" Jul 02 19:41:54.713057 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:54.712914 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06530fb0-aeed-4c86-84dd-4c66edc1bbce-openshift-service-ca\") pod \"maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f\" (UID: \"06530fb0-aeed-4c86-84dd-4c66edc1bbce\") " pod="opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" Jul 02 19:41:54.713057 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:41:54.712972 2562 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-5a2a19-serving-cert: secret "maas-api-e2e-api-b-5a2a19-serving-cert" not found Jul 02 19:41:54.713057 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:41:54.713046 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06530fb0-aeed-4c86-84dd-4c66edc1bbce-maas-api-tls podName:06530fb0-aeed-4c86-84dd-4c66edc1bbce nodeName:}" failed. No retries permitted until 2026-07-02 19:41:55.213024364 +0000 UTC m=+2022.990041918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/06530fb0-aeed-4c86-84dd-4c66edc1bbce-maas-api-tls") pod "maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" (UID: "06530fb0-aeed-4c86-84dd-4c66edc1bbce") : secret "maas-api-e2e-api-b-5a2a19-serving-cert" not found Jul 02 19:41:54.713538 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:54.713519 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06530fb0-aeed-4c86-84dd-4c66edc1bbce-openshift-service-ca\") pod \"maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f\" (UID: \"06530fb0-aeed-4c86-84dd-4c66edc1bbce\") " pod="opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" Jul 02 19:41:54.723829 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:54.723802 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck479\" (UniqueName: \"kubernetes.io/projected/06530fb0-aeed-4c86-84dd-4c66edc1bbce-kube-api-access-ck479\") pod \"maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f\" (UID: \"06530fb0-aeed-4c86-84dd-4c66edc1bbce\") " pod="opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" Jul 02 19:41:54.771011 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:54.770979 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:55.203419 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:55.203379 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:55.217190 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:55.217160 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06530fb0-aeed-4c86-84dd-4c66edc1bbce-maas-api-tls\") pod \"maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f\" (UID: \"06530fb0-aeed-4c86-84dd-4c66edc1bbce\") " pod="opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" Jul 02 19:41:55.219870 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:55.219845 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06530fb0-aeed-4c86-84dd-4c66edc1bbce-maas-api-tls\") pod \"maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f\" (UID: \"06530fb0-aeed-4c86-84dd-4c66edc1bbce\") " pod="opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" Jul 02 19:41:55.301700 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:55.301669 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:55.421898 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:55.421869 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" Jul 02 19:41:55.567222 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:55.567198 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f"] Jul 02 19:41:55.569634 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:41:55.569564 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06530fb0_aeed_4c86_84dd_4c66edc1bbce.slice/crio-aa5be756692b45134b7e77eda45d3fb384a7eb0eed55eb0961a4abb5fac122aa WatchSource:0}: Error finding container aa5be756692b45134b7e77eda45d3fb384a7eb0eed55eb0961a4abb5fac122aa: Status 404 returned error can't find the container with id aa5be756692b45134b7e77eda45d3fb384a7eb0eed55eb0961a4abb5fac122aa Jul 02 19:41:55.813328 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:55.813283 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" event={"ID":"06530fb0-aeed-4c86-84dd-4c66edc1bbce","Type":"ContainerStarted","Data":"aa5be756692b45134b7e77eda45d3fb384a7eb0eed55eb0961a4abb5fac122aa"} Jul 02 19:41:56.819954 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:56.819918 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" event={"ID":"06530fb0-aeed-4c86-84dd-4c66edc1bbce","Type":"ContainerStarted","Data":"bb01f7cd13632e5f25c12a4b25253224af72eb8ecf2a8566bfdd9fba5caac6d1"} Jul 02 19:41:56.820401 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:56.820061 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" Jul 02 19:41:56.837936 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:56.837885 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" podStartSLOduration=2.348493945 podStartE2EDuration="2.837870439s" totalImagesPullingTime="489.376494ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:41:56.83636846 +0000 UTC m=+2024.613386052" watchObservedRunningTime="2026-07-02 19:41:56.837870439 +0000 UTC m=+2024.614888006" Jul 02 19:41:58.953533 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:41:58.953499 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:02.831144 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:02.831110 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" Jul 02 19:42:05.714892 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:05.714858 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:05.815796 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:05.815754 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:07.107089 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.107055 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:07.218040 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.217994 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:07.380897 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.380826 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c"] Jul 02 19:42:07.381048 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.381026 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c" podUID="573b76fd-5bb0-4658-8426-ef85cd44d383" containerName="maas-api" containerID="cri-o://ce33e19034ce696398a77705ab6b0cef623d01afd4d614f09d6fc9aad961e107" gracePeriod=30 Jul 02 19:42:07.428183 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.428152 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:07.640705 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.640647 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c" Jul 02 19:42:07.641860 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.641835 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:07.738764 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.738732 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/573b76fd-5bb0-4658-8426-ef85cd44d383-openshift-service-ca\") pod \"573b76fd-5bb0-4658-8426-ef85cd44d383\" (UID: \"573b76fd-5bb0-4658-8426-ef85cd44d383\") " Jul 02 19:42:07.738933 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.738809 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/573b76fd-5bb0-4658-8426-ef85cd44d383-maas-api-tls\") pod \"573b76fd-5bb0-4658-8426-ef85cd44d383\" (UID: \"573b76fd-5bb0-4658-8426-ef85cd44d383\") " Jul 02 19:42:07.738933 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.738839 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfws8\" (UniqueName: \"kubernetes.io/projected/573b76fd-5bb0-4658-8426-ef85cd44d383-kube-api-access-jfws8\") pod \"573b76fd-5bb0-4658-8426-ef85cd44d383\" (UID: \"573b76fd-5bb0-4658-8426-ef85cd44d383\") " Jul 02 19:42:07.739115 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.739091 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/573b76fd-5bb0-4658-8426-ef85cd44d383-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "573b76fd-5bb0-4658-8426-ef85cd44d383" (UID: "573b76fd-5bb0-4658-8426-ef85cd44d383"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:42:07.739188 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.739177 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/573b76fd-5bb0-4658-8426-ef85cd44d383-openshift-service-ca\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:42:07.741125 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.741098 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/573b76fd-5bb0-4658-8426-ef85cd44d383-kube-api-access-jfws8" (OuterVolumeSpecName: "kube-api-access-jfws8") pod "573b76fd-5bb0-4658-8426-ef85cd44d383" (UID: "573b76fd-5bb0-4658-8426-ef85cd44d383"). InnerVolumeSpecName "kube-api-access-jfws8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:42:07.741255 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.741179 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/573b76fd-5bb0-4658-8426-ef85cd44d383-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "573b76fd-5bb0-4658-8426-ef85cd44d383" (UID: "573b76fd-5bb0-4658-8426-ef85cd44d383"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:42:07.840523 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.840496 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/573b76fd-5bb0-4658-8426-ef85cd44d383-maas-api-tls\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:42:07.840662 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.840528 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfws8\" (UniqueName: \"kubernetes.io/projected/573b76fd-5bb0-4658-8426-ef85cd44d383-kube-api-access-jfws8\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:42:07.876742 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.876713 2562 generic.go:356] "Generic (PLEG): container finished" podID="573b76fd-5bb0-4658-8426-ef85cd44d383" containerID="ce33e19034ce696398a77705ab6b0cef623d01afd4d614f09d6fc9aad961e107" exitCode=0 Jul 02 19:42:07.876842 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.876756 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c" Jul 02 19:42:07.876842 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.876788 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c" event={"ID":"573b76fd-5bb0-4658-8426-ef85cd44d383","Type":"ContainerDied","Data":"ce33e19034ce696398a77705ab6b0cef623d01afd4d614f09d6fc9aad961e107"} Jul 02 19:42:07.876842 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.876830 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c" event={"ID":"573b76fd-5bb0-4658-8426-ef85cd44d383","Type":"ContainerDied","Data":"893de9abf510436174ced0ead1b98f512c75775adeedf053f49b0546ea4e5cef"} Jul 02 19:42:07.876957 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.876850 2562 scope.go:117] "RemoveContainer" containerID="ce33e19034ce696398a77705ab6b0cef623d01afd4d614f09d6fc9aad961e107" Jul 02 19:42:07.885667 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.885650 2562 scope.go:117] "RemoveContainer" containerID="ce33e19034ce696398a77705ab6b0cef623d01afd4d614f09d6fc9aad961e107" Jul 02 19:42:07.885925 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:42:07.885905 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce33e19034ce696398a77705ab6b0cef623d01afd4d614f09d6fc9aad961e107\": container with ID starting with ce33e19034ce696398a77705ab6b0cef623d01afd4d614f09d6fc9aad961e107 not found: ID does not exist" containerID="ce33e19034ce696398a77705ab6b0cef623d01afd4d614f09d6fc9aad961e107" Jul 02 19:42:07.885991 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.885932 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce33e19034ce696398a77705ab6b0cef623d01afd4d614f09d6fc9aad961e107"} err="failed to get container status \"ce33e19034ce696398a77705ab6b0cef623d01afd4d614f09d6fc9aad961e107\": rpc error: code = NotFound desc = could not find container \"ce33e19034ce696398a77705ab6b0cef623d01afd4d614f09d6fc9aad961e107\": container with ID starting with ce33e19034ce696398a77705ab6b0cef623d01afd4d614f09d6fc9aad961e107 not found: ID does not exist" Jul 02 19:42:07.913956 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.913891 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c"] Jul 02 19:42:07.917070 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:07.917049 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-a91b31-6555b464f5-ntp8c"] Jul 02 19:42:08.776914 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:08.776883 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="573b76fd-5bb0-4658-8426-ef85cd44d383" path="/var/lib/kubelet/pods/573b76fd-5bb0-4658-8426-ef85cd44d383/volumes" Jul 02 19:42:13.373341 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:13.373306 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:13.479633 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:13.479579 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:13.664257 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:13.664173 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:13.888015 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:13.887984 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:14.080229 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:14.080195 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:14.529318 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:14.529287 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:14.696068 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:14.696034 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:14.798574 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:14.798493 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:14.922140 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:14.922105 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f"] Jul 02 19:42:14.922373 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:14.922347 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" podUID="06530fb0-aeed-4c86-84dd-4c66edc1bbce" containerName="maas-api" containerID="cri-o://bb01f7cd13632e5f25c12a4b25253224af72eb8ecf2a8566bfdd9fba5caac6d1" gracePeriod=30 Jul 02 19:42:14.962448 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:14.962416 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:15.173841 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:15.173815 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" Jul 02 19:42:15.214328 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:15.214303 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06530fb0-aeed-4c86-84dd-4c66edc1bbce-openshift-service-ca\") pod \"06530fb0-aeed-4c86-84dd-4c66edc1bbce\" (UID: \"06530fb0-aeed-4c86-84dd-4c66edc1bbce\") " Jul 02 19:42:15.214465 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:15.214385 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ck479\" (UniqueName: \"kubernetes.io/projected/06530fb0-aeed-4c86-84dd-4c66edc1bbce-kube-api-access-ck479\") pod \"06530fb0-aeed-4c86-84dd-4c66edc1bbce\" (UID: \"06530fb0-aeed-4c86-84dd-4c66edc1bbce\") " Jul 02 19:42:15.214505 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:15.214491 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06530fb0-aeed-4c86-84dd-4c66edc1bbce-maas-api-tls\") pod \"06530fb0-aeed-4c86-84dd-4c66edc1bbce\" (UID: \"06530fb0-aeed-4c86-84dd-4c66edc1bbce\") " Jul 02 19:42:15.214738 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:15.214709 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06530fb0-aeed-4c86-84dd-4c66edc1bbce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "06530fb0-aeed-4c86-84dd-4c66edc1bbce" (UID: "06530fb0-aeed-4c86-84dd-4c66edc1bbce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:42:15.216484 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:15.216457 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06530fb0-aeed-4c86-84dd-4c66edc1bbce-kube-api-access-ck479" (OuterVolumeSpecName: "kube-api-access-ck479") pod "06530fb0-aeed-4c86-84dd-4c66edc1bbce" (UID: "06530fb0-aeed-4c86-84dd-4c66edc1bbce"). InnerVolumeSpecName "kube-api-access-ck479". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:42:15.216679 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:15.216652 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06530fb0-aeed-4c86-84dd-4c66edc1bbce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "06530fb0-aeed-4c86-84dd-4c66edc1bbce" (UID: "06530fb0-aeed-4c86-84dd-4c66edc1bbce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:42:15.315447 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:15.315420 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06530fb0-aeed-4c86-84dd-4c66edc1bbce-openshift-service-ca\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:42:15.315447 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:15.315445 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck479\" (UniqueName: \"kubernetes.io/projected/06530fb0-aeed-4c86-84dd-4c66edc1bbce-kube-api-access-ck479\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:42:15.315628 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:15.315456 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06530fb0-aeed-4c86-84dd-4c66edc1bbce-maas-api-tls\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:42:15.920260 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:15.920230 2562 generic.go:356] "Generic (PLEG): container finished" podID="06530fb0-aeed-4c86-84dd-4c66edc1bbce" containerID="bb01f7cd13632e5f25c12a4b25253224af72eb8ecf2a8566bfdd9fba5caac6d1" exitCode=0 Jul 02 19:42:15.920724 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:15.920278 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" Jul 02 19:42:15.920724 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:15.920322 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" event={"ID":"06530fb0-aeed-4c86-84dd-4c66edc1bbce","Type":"ContainerDied","Data":"bb01f7cd13632e5f25c12a4b25253224af72eb8ecf2a8566bfdd9fba5caac6d1"} Jul 02 19:42:15.920724 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:15.920377 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f" event={"ID":"06530fb0-aeed-4c86-84dd-4c66edc1bbce","Type":"ContainerDied","Data":"aa5be756692b45134b7e77eda45d3fb384a7eb0eed55eb0961a4abb5fac122aa"} Jul 02 19:42:15.920724 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:15.920406 2562 scope.go:117] "RemoveContainer" containerID="bb01f7cd13632e5f25c12a4b25253224af72eb8ecf2a8566bfdd9fba5caac6d1" Jul 02 19:42:15.930250 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:15.929983 2562 scope.go:117] "RemoveContainer" containerID="bb01f7cd13632e5f25c12a4b25253224af72eb8ecf2a8566bfdd9fba5caac6d1" Jul 02 19:42:15.930322 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:42:15.930277 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb01f7cd13632e5f25c12a4b25253224af72eb8ecf2a8566bfdd9fba5caac6d1\": container with ID starting with bb01f7cd13632e5f25c12a4b25253224af72eb8ecf2a8566bfdd9fba5caac6d1 not found: ID does not exist" containerID="bb01f7cd13632e5f25c12a4b25253224af72eb8ecf2a8566bfdd9fba5caac6d1" Jul 02 19:42:15.930322 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:15.930309 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb01f7cd13632e5f25c12a4b25253224af72eb8ecf2a8566bfdd9fba5caac6d1"} err="failed to get container status \"bb01f7cd13632e5f25c12a4b25253224af72eb8ecf2a8566bfdd9fba5caac6d1\": rpc error: code = NotFound desc = could not find container \"bb01f7cd13632e5f25c12a4b25253224af72eb8ecf2a8566bfdd9fba5caac6d1\": container with ID starting with bb01f7cd13632e5f25c12a4b25253224af72eb8ecf2a8566bfdd9fba5caac6d1 not found: ID does not exist" Jul 02 19:42:15.944347 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:15.944321 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f"] Jul 02 19:42:15.954449 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:15.954423 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-5a2a19-7974fd5786-5ln5f"] Jul 02 19:42:16.777913 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:16.777883 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06530fb0-aeed-4c86-84dd-4c66edc1bbce" path="/var/lib/kubelet/pods/06530fb0-aeed-4c86-84dd-4c66edc1bbce/volumes" Jul 02 19:42:20.497838 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:20.497807 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:20.598052 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:20.598015 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:20.778450 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:20.778414 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:21.096231 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:21.096149 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:21.368310 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:21.368230 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:21.686455 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:21.686378 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:21.905407 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:21.905377 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:24.649099 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.649064 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:24.671205 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.671172 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n"] Jul 02 19:42:24.671846 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.671825 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06530fb0-aeed-4c86-84dd-4c66edc1bbce" containerName="maas-api" Jul 02 19:42:24.671951 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.671850 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="06530fb0-aeed-4c86-84dd-4c66edc1bbce" containerName="maas-api" Jul 02 19:42:24.671951 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.671901 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="573b76fd-5bb0-4658-8426-ef85cd44d383" containerName="maas-api" Jul 02 19:42:24.671951 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.671911 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="573b76fd-5bb0-4658-8426-ef85cd44d383" containerName="maas-api" Jul 02 19:42:24.672117 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.672003 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="573b76fd-5bb0-4658-8426-ef85cd44d383" containerName="maas-api" Jul 02 19:42:24.672117 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.672019 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="06530fb0-aeed-4c86-84dd-4c66edc1bbce" containerName="maas-api" Jul 02 19:42:24.677457 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.677431 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n" Jul 02 19:42:24.680851 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.680832 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-67cf22-serving-cert\"" Jul 02 19:42:24.687581 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.687551 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n"] Jul 02 19:42:24.753231 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.753191 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:24.803022 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.802993 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9beb6e93-7821-4ade-91e4-adf74805fb87-maas-api-tls\") pod \"maas-api-e2e-shared-a-67cf22-6688458545-bc54n\" (UID: \"9beb6e93-7821-4ade-91e4-adf74805fb87\") " pod="opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n" Jul 02 19:42:24.803022 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.803026 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8hts\" (UniqueName: \"kubernetes.io/projected/9beb6e93-7821-4ade-91e4-adf74805fb87-kube-api-access-h8hts\") pod \"maas-api-e2e-shared-a-67cf22-6688458545-bc54n\" (UID: \"9beb6e93-7821-4ade-91e4-adf74805fb87\") " pod="opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n" Jul 02 19:42:24.803248 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.803061 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9beb6e93-7821-4ade-91e4-adf74805fb87-openshift-service-ca\") pod \"maas-api-e2e-shared-a-67cf22-6688458545-bc54n\" (UID: \"9beb6e93-7821-4ade-91e4-adf74805fb87\") " pod="opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n" Jul 02 19:42:24.904368 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.904284 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9beb6e93-7821-4ade-91e4-adf74805fb87-maas-api-tls\") pod \"maas-api-e2e-shared-a-67cf22-6688458545-bc54n\" (UID: \"9beb6e93-7821-4ade-91e4-adf74805fb87\") " pod="opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n" Jul 02 19:42:24.904368 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.904315 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8hts\" (UniqueName: \"kubernetes.io/projected/9beb6e93-7821-4ade-91e4-adf74805fb87-kube-api-access-h8hts\") pod \"maas-api-e2e-shared-a-67cf22-6688458545-bc54n\" (UID: \"9beb6e93-7821-4ade-91e4-adf74805fb87\") " pod="opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n" Jul 02 19:42:24.904586 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.904445 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9beb6e93-7821-4ade-91e4-adf74805fb87-openshift-service-ca\") pod \"maas-api-e2e-shared-a-67cf22-6688458545-bc54n\" (UID: \"9beb6e93-7821-4ade-91e4-adf74805fb87\") " pod="opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n" Jul 02 19:42:24.905084 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.905056 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9beb6e93-7821-4ade-91e4-adf74805fb87-openshift-service-ca\") pod \"maas-api-e2e-shared-a-67cf22-6688458545-bc54n\" (UID: \"9beb6e93-7821-4ade-91e4-adf74805fb87\") " pod="opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n" Jul 02 19:42:24.906910 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.906888 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9beb6e93-7821-4ade-91e4-adf74805fb87-maas-api-tls\") pod \"maas-api-e2e-shared-a-67cf22-6688458545-bc54n\" (UID: \"9beb6e93-7821-4ade-91e4-adf74805fb87\") " pod="opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n" Jul 02 19:42:24.914061 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.914036 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8hts\" (UniqueName: \"kubernetes.io/projected/9beb6e93-7821-4ade-91e4-adf74805fb87-kube-api-access-h8hts\") pod \"maas-api-e2e-shared-a-67cf22-6688458545-bc54n\" (UID: \"9beb6e93-7821-4ade-91e4-adf74805fb87\") " pod="opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n" Jul 02 19:42:24.938378 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.938344 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:24.989330 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:24.989299 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n" Jul 02 19:42:25.135345 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:25.135258 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n"] Jul 02 19:42:25.137678 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:42:25.137642 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9beb6e93_7821_4ade_91e4_adf74805fb87.slice/crio-da528435decbe0fec017f4ad5df8ee7486242fb4b958e511bb38a4fc576ca264 WatchSource:0}: Error finding container da528435decbe0fec017f4ad5df8ee7486242fb4b958e511bb38a4fc576ca264: Status 404 returned error can't find the container with id da528435decbe0fec017f4ad5df8ee7486242fb4b958e511bb38a4fc576ca264 Jul 02 19:42:25.184782 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:25.184719 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:25.284315 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:25.284284 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:25.977662 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:25.977562 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n" event={"ID":"9beb6e93-7821-4ade-91e4-adf74805fb87","Type":"ContainerStarted","Data":"cfa5472b48491f99cb7e4ec9782591a8b9c1477f2c09998e1b1f4d7d13206365"} Jul 02 19:42:25.978008 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:25.977742 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n" Jul 02 19:42:25.978008 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:25.977777 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n" event={"ID":"9beb6e93-7821-4ade-91e4-adf74805fb87","Type":"ContainerStarted","Data":"da528435decbe0fec017f4ad5df8ee7486242fb4b958e511bb38a4fc576ca264"} Jul 02 19:42:25.997137 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:25.997093 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n" podStartSLOduration=1.5501224740000001 podStartE2EDuration="1.997080075s" totalImagesPullingTime="446.957601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:42:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:42:25.995824839 +0000 UTC m=+2053.772842429" watchObservedRunningTime="2026-07-02 19:42:25.997080075 +0000 UTC m=+2053.774097661" Jul 02 19:42:29.137569 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:29.137536 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:29.474565 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:29.474470 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:29.869343 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:29.869307 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:30.329248 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:30.329205 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:31.987742 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:31.987706 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n" Jul 02 19:42:32.654690 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:32.654576 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:32.766579 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:32.766541 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:32.870844 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:32.870812 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb"] Jul 02 19:42:32.874920 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:32.874897 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb" Jul 02 19:42:32.877619 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:32.877579 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-0c31dc-serving-cert\"" Jul 02 19:42:32.887197 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:32.887173 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb"] Jul 02 19:42:32.975389 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:32.975317 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:32.981240 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:32.981220 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d8eef16-f595-41be-9e77-e61a0235d6d9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb\" (UID: \"4d8eef16-f595-41be-9e77-e61a0235d6d9\") " pod="opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb" Jul 02 19:42:32.981312 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:32.981298 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsrgj\" (UniqueName: \"kubernetes.io/projected/4d8eef16-f595-41be-9e77-e61a0235d6d9-kube-api-access-qsrgj\") pod \"maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb\" (UID: \"4d8eef16-f595-41be-9e77-e61a0235d6d9\") " pod="opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb" Jul 02 19:42:32.981361 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:32.981322 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d8eef16-f595-41be-9e77-e61a0235d6d9-maas-api-tls\") pod \"maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb\" (UID: \"4d8eef16-f595-41be-9e77-e61a0235d6d9\") " pod="opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb" Jul 02 19:42:33.082014 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:33.081984 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsrgj\" (UniqueName: \"kubernetes.io/projected/4d8eef16-f595-41be-9e77-e61a0235d6d9-kube-api-access-qsrgj\") pod \"maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb\" (UID: \"4d8eef16-f595-41be-9e77-e61a0235d6d9\") " pod="opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb" Jul 02 19:42:33.082014 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:33.082017 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d8eef16-f595-41be-9e77-e61a0235d6d9-maas-api-tls\") pod \"maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb\" (UID: \"4d8eef16-f595-41be-9e77-e61a0235d6d9\") " pod="opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb" Jul 02 19:42:33.082458 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:33.082074 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d8eef16-f595-41be-9e77-e61a0235d6d9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb\" (UID: \"4d8eef16-f595-41be-9e77-e61a0235d6d9\") " pod="opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb" Jul 02 19:42:33.082744 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:33.082720 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d8eef16-f595-41be-9e77-e61a0235d6d9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb\" (UID: \"4d8eef16-f595-41be-9e77-e61a0235d6d9\") " pod="opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb" Jul 02 19:42:33.084514 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:33.084491 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d8eef16-f595-41be-9e77-e61a0235d6d9-maas-api-tls\") pod \"maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb\" (UID: \"4d8eef16-f595-41be-9e77-e61a0235d6d9\") " pod="opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb" Jul 02 19:42:33.092101 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:33.092078 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsrgj\" (UniqueName: \"kubernetes.io/projected/4d8eef16-f595-41be-9e77-e61a0235d6d9-kube-api-access-qsrgj\") pod \"maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb\" (UID: \"4d8eef16-f595-41be-9e77-e61a0235d6d9\") " pod="opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb" Jul 02 19:42:33.097341 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:33.097311 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:33.187450 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:33.187428 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb" Jul 02 19:42:33.337830 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:33.337798 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb"] Jul 02 19:42:33.509081 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:33.508979 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:33.605319 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:33.605286 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:34.019488 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:34.019456 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb" event={"ID":"4d8eef16-f595-41be-9e77-e61a0235d6d9","Type":"ContainerStarted","Data":"ed7fd346bb2a07c685cf5bbb6df4a6485eca5d778d87753d8bdee7ceef1e2c85"} Jul 02 19:42:34.019674 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:34.019503 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb" event={"ID":"4d8eef16-f595-41be-9e77-e61a0235d6d9","Type":"ContainerStarted","Data":"d1076806b5296394e52835d99cc35b5f4d7660767acf73523b5ecf0cf4ef3d31"} Jul 02 19:42:34.019674 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:34.019525 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb" Jul 02 19:42:34.040068 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:34.040024 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb" podStartSLOduration=1.715581255 podStartE2EDuration="2.040009552s" totalImagesPullingTime="324.428297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:42:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:42:34.03770501 +0000 UTC m=+2061.814722601" watchObservedRunningTime="2026-07-02 19:42:34.040009552 +0000 UTC m=+2061.817027119" Jul 02 19:42:37.190414 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:37.190364 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:40.029997 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:40.029969 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb" Jul 02 19:42:45.884139 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:45.884105 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:45.985111 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:45.985079 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:50.122846 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:50.122815 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:54.979392 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:54.979332 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:55.079748 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:42:55.079713 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:43:24.014233 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:43:24.014182 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:43:36.097559 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:43:36.097516 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:44:40.118195 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:44:40.118157 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:45:00.133699 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:00.133659 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29717025-qrrfl"] Jul 02 19:45:00.137341 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:00.137323 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717025-qrrfl" Jul 02 19:45:00.146108 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:00.146076 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29717025-qrrfl"] Jul 02 19:45:00.320718 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:00.320683 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjxxl\" (UniqueName: \"kubernetes.io/projected/3716bd2d-c17e-4613-ab71-c78b272606fa-kube-api-access-sjxxl\") pod \"maas-api-key-cleanup-29717025-qrrfl\" (UID: \"3716bd2d-c17e-4613-ab71-c78b272606fa\") " pod="opendatahub/maas-api-key-cleanup-29717025-qrrfl" Jul 02 19:45:00.421277 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:00.421208 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjxxl\" (UniqueName: \"kubernetes.io/projected/3716bd2d-c17e-4613-ab71-c78b272606fa-kube-api-access-sjxxl\") pod \"maas-api-key-cleanup-29717025-qrrfl\" (UID: \"3716bd2d-c17e-4613-ab71-c78b272606fa\") " pod="opendatahub/maas-api-key-cleanup-29717025-qrrfl" Jul 02 19:45:00.429931 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:00.429904 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjxxl\" (UniqueName: \"kubernetes.io/projected/3716bd2d-c17e-4613-ab71-c78b272606fa-kube-api-access-sjxxl\") pod \"maas-api-key-cleanup-29717025-qrrfl\" (UID: \"3716bd2d-c17e-4613-ab71-c78b272606fa\") " pod="opendatahub/maas-api-key-cleanup-29717025-qrrfl" Jul 02 19:45:00.450643 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:00.450624 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717025-qrrfl" Jul 02 19:45:00.789352 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:00.789292 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29717025-qrrfl"] Jul 02 19:45:00.791951 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:45:00.791923 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3716bd2d_c17e_4613_ab71_c78b272606fa.slice/crio-e42dd5a6f8e1876842271eebc4ec31d3c404778a56507089e6843d4f2ef76b23 WatchSource:0}: Error finding container e42dd5a6f8e1876842271eebc4ec31d3c404778a56507089e6843d4f2ef76b23: Status 404 returned error can't find the container with id e42dd5a6f8e1876842271eebc4ec31d3c404778a56507089e6843d4f2ef76b23 Jul 02 19:45:01.779186 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:01.779159 2562 generic.go:356] "Generic (PLEG): container finished" podID="3716bd2d-c17e-4613-ab71-c78b272606fa" containerID="4ac7a437c66e8ae8efe08fb98921bb2d9b737644bde1ca579193cce70cb0152e" exitCode=0 Jul 02 19:45:01.779471 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:01.779196 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717025-qrrfl" event={"ID":"3716bd2d-c17e-4613-ab71-c78b272606fa","Type":"ContainerDied","Data":"4ac7a437c66e8ae8efe08fb98921bb2d9b737644bde1ca579193cce70cb0152e"} Jul 02 19:45:01.779471 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:01.779233 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717025-qrrfl" event={"ID":"3716bd2d-c17e-4613-ab71-c78b272606fa","Type":"ContainerStarted","Data":"e42dd5a6f8e1876842271eebc4ec31d3c404778a56507089e6843d4f2ef76b23"} Jul 02 19:45:02.917871 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:02.917846 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717025-qrrfl" Jul 02 19:45:02.945218 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:02.945194 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjxxl\" (UniqueName: \"kubernetes.io/projected/3716bd2d-c17e-4613-ab71-c78b272606fa-kube-api-access-sjxxl\") pod \"3716bd2d-c17e-4613-ab71-c78b272606fa\" (UID: \"3716bd2d-c17e-4613-ab71-c78b272606fa\") " Jul 02 19:45:02.947278 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:02.947256 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3716bd2d-c17e-4613-ab71-c78b272606fa-kube-api-access-sjxxl" (OuterVolumeSpecName: "kube-api-access-sjxxl") pod "3716bd2d-c17e-4613-ab71-c78b272606fa" (UID: "3716bd2d-c17e-4613-ab71-c78b272606fa"). InnerVolumeSpecName "kube-api-access-sjxxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:45:03.046832 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:03.046764 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjxxl\" (UniqueName: \"kubernetes.io/projected/3716bd2d-c17e-4613-ab71-c78b272606fa-kube-api-access-sjxxl\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:45:03.788824 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:03.788790 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717025-qrrfl" Jul 02 19:45:03.788824 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:03.788812 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717025-qrrfl" event={"ID":"3716bd2d-c17e-4613-ab71-c78b272606fa","Type":"ContainerDied","Data":"e42dd5a6f8e1876842271eebc4ec31d3c404778a56507089e6843d4f2ef76b23"} Jul 02 19:45:03.789014 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:03.788849 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e42dd5a6f8e1876842271eebc4ec31d3c404778a56507089e6843d4f2ef76b23" Jul 02 19:45:39.198478 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:39.198404 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:45:40.200801 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:40.200765 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:45:46.346987 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:46.346940 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:45:46.403866 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:46.403833 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:45:52.091354 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:45:52.091320 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:46:10.907118 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:46:10.907081 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:46:14.908235 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:46:14.908197 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:47:02.445434 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:47:02.445393 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:47:03.621384 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:47:03.621350 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:47:06.482941 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:47:06.482905 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:47:06.585484 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:47:06.585449 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:47:10.141903 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:47:10.141872 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:47:43.923290 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:47:43.923253 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:47:46.635287 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:47:46.635252 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:47:46.735899 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:47:46.735865 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:47:55.711059 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:47:55.711019 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:47:55.808483 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:47:55.808438 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:48:23.017159 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:23.017111 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:48:23.884013 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:23.883974 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:48:25.081640 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:25.081585 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:48:25.477195 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:25.477103 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:48:25.601278 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:25.601243 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:48:25.737468 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:25.737366 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:48:25.852577 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:25.852548 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n"] Jul 02 19:48:25.852794 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:25.852769 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n" podUID="9beb6e93-7821-4ade-91e4-adf74805fb87" containerName="maas-api" containerID="cri-o://cfa5472b48491f99cb7e4ec9782591a8b9c1477f2c09998e1b1f4d7d13206365" gracePeriod=30 Jul 02 19:48:25.896472 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:25.896434 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:48:26.113122 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:26.113100 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n" Jul 02 19:48:26.205904 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:26.205868 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9beb6e93-7821-4ade-91e4-adf74805fb87-maas-api-tls\") pod \"9beb6e93-7821-4ade-91e4-adf74805fb87\" (UID: \"9beb6e93-7821-4ade-91e4-adf74805fb87\") " Jul 02 19:48:26.206039 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:26.205969 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8hts\" (UniqueName: \"kubernetes.io/projected/9beb6e93-7821-4ade-91e4-adf74805fb87-kube-api-access-h8hts\") pod \"9beb6e93-7821-4ade-91e4-adf74805fb87\" (UID: \"9beb6e93-7821-4ade-91e4-adf74805fb87\") " Jul 02 19:48:26.208099 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:26.208068 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9beb6e93-7821-4ade-91e4-adf74805fb87-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9beb6e93-7821-4ade-91e4-adf74805fb87" (UID: "9beb6e93-7821-4ade-91e4-adf74805fb87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:48:26.208197 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:26.208098 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9beb6e93-7821-4ade-91e4-adf74805fb87-kube-api-access-h8hts" (OuterVolumeSpecName: "kube-api-access-h8hts") pod "9beb6e93-7821-4ade-91e4-adf74805fb87" (UID: "9beb6e93-7821-4ade-91e4-adf74805fb87"). InnerVolumeSpecName "kube-api-access-h8hts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:48:26.306743 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:26.306680 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9beb6e93-7821-4ade-91e4-adf74805fb87-openshift-service-ca\") pod \"9beb6e93-7821-4ade-91e4-adf74805fb87\" (UID: \"9beb6e93-7821-4ade-91e4-adf74805fb87\") " Jul 02 19:48:26.307052 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:26.307032 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9beb6e93-7821-4ade-91e4-adf74805fb87-maas-api-tls\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:48:26.307095 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:26.307067 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8hts\" (UniqueName: \"kubernetes.io/projected/9beb6e93-7821-4ade-91e4-adf74805fb87-kube-api-access-h8hts\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:48:26.307095 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:26.307030 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9beb6e93-7821-4ade-91e4-adf74805fb87-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9beb6e93-7821-4ade-91e4-adf74805fb87" (UID: "9beb6e93-7821-4ade-91e4-adf74805fb87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:48:26.407910 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:26.407884 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9beb6e93-7821-4ade-91e4-adf74805fb87-openshift-service-ca\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:48:26.836654 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:26.836625 2562 generic.go:356] "Generic (PLEG): container finished" podID="9beb6e93-7821-4ade-91e4-adf74805fb87" containerID="cfa5472b48491f99cb7e4ec9782591a8b9c1477f2c09998e1b1f4d7d13206365" exitCode=0 Jul 02 19:48:26.836784 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:26.836701 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n" Jul 02 19:48:26.836830 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:26.836698 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n" event={"ID":"9beb6e93-7821-4ade-91e4-adf74805fb87","Type":"ContainerDied","Data":"cfa5472b48491f99cb7e4ec9782591a8b9c1477f2c09998e1b1f4d7d13206365"} Jul 02 19:48:26.836872 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:26.836825 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n" event={"ID":"9beb6e93-7821-4ade-91e4-adf74805fb87","Type":"ContainerDied","Data":"da528435decbe0fec017f4ad5df8ee7486242fb4b958e511bb38a4fc576ca264"} Jul 02 19:48:26.836872 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:26.836854 2562 scope.go:117] "RemoveContainer" containerID="cfa5472b48491f99cb7e4ec9782591a8b9c1477f2c09998e1b1f4d7d13206365" Jul 02 19:48:26.845681 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:26.845658 2562 scope.go:117] "RemoveContainer" containerID="cfa5472b48491f99cb7e4ec9782591a8b9c1477f2c09998e1b1f4d7d13206365" Jul 02 19:48:26.845943 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:48:26.845904 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfa5472b48491f99cb7e4ec9782591a8b9c1477f2c09998e1b1f4d7d13206365\": container with ID starting with cfa5472b48491f99cb7e4ec9782591a8b9c1477f2c09998e1b1f4d7d13206365 not found: ID does not exist" containerID="cfa5472b48491f99cb7e4ec9782591a8b9c1477f2c09998e1b1f4d7d13206365" Jul 02 19:48:26.845943 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:26.845931 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfa5472b48491f99cb7e4ec9782591a8b9c1477f2c09998e1b1f4d7d13206365"} err="failed to get container status \"cfa5472b48491f99cb7e4ec9782591a8b9c1477f2c09998e1b1f4d7d13206365\": rpc error: code = NotFound desc = could not find container \"cfa5472b48491f99cb7e4ec9782591a8b9c1477f2c09998e1b1f4d7d13206365\": container with ID starting with cfa5472b48491f99cb7e4ec9782591a8b9c1477f2c09998e1b1f4d7d13206365 not found: ID does not exist" Jul 02 19:48:26.856579 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:26.856558 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n"] Jul 02 19:48:26.862901 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:26.862880 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-67cf22-6688458545-bc54n"] Jul 02 19:48:28.777153 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:28.777118 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9beb6e93-7821-4ade-91e4-adf74805fb87" path="/var/lib/kubelet/pods/9beb6e93-7821-4ade-91e4-adf74805fb87/volumes" Jul 02 19:48:30.525009 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:30.524978 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:48:53.894145 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:53.894042 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:48:54.298478 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:54.298431 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:48:54.486430 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:54.486395 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:48:54.634760 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:54.634673 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:48:55.031499 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:55.031469 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:48:55.135294 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:55.135253 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:48:55.259101 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:55.259067 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb"] Jul 02 19:48:55.259396 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:55.259348 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb" podUID="4d8eef16-f595-41be-9e77-e61a0235d6d9" containerName="maas-api" containerID="cri-o://ed7fd346bb2a07c685cf5bbb6df4a6485eca5d778d87753d8bdee7ceef1e2c85" gracePeriod=30 Jul 02 19:48:55.283865 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:55.283785 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:48:55.715984 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:55.715957 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb" Jul 02 19:48:55.854549 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:55.854476 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d8eef16-f595-41be-9e77-e61a0235d6d9-maas-api-tls\") pod \"4d8eef16-f595-41be-9e77-e61a0235d6d9\" (UID: \"4d8eef16-f595-41be-9e77-e61a0235d6d9\") " Jul 02 19:48:55.854549 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:55.854545 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d8eef16-f595-41be-9e77-e61a0235d6d9-openshift-service-ca\") pod \"4d8eef16-f595-41be-9e77-e61a0235d6d9\" (UID: \"4d8eef16-f595-41be-9e77-e61a0235d6d9\") " Jul 02 19:48:55.854748 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:55.854620 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsrgj\" (UniqueName: \"kubernetes.io/projected/4d8eef16-f595-41be-9e77-e61a0235d6d9-kube-api-access-qsrgj\") pod \"4d8eef16-f595-41be-9e77-e61a0235d6d9\" (UID: \"4d8eef16-f595-41be-9e77-e61a0235d6d9\") " Jul 02 19:48:55.854873 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:55.854849 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d8eef16-f595-41be-9e77-e61a0235d6d9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4d8eef16-f595-41be-9e77-e61a0235d6d9" (UID: "4d8eef16-f595-41be-9e77-e61a0235d6d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:48:55.855398 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:55.855021 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d8eef16-f595-41be-9e77-e61a0235d6d9-openshift-service-ca\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:48:55.856828 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:55.856805 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d8eef16-f595-41be-9e77-e61a0235d6d9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4d8eef16-f595-41be-9e77-e61a0235d6d9" (UID: "4d8eef16-f595-41be-9e77-e61a0235d6d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:48:55.856932 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:55.856860 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8eef16-f595-41be-9e77-e61a0235d6d9-kube-api-access-qsrgj" (OuterVolumeSpecName: "kube-api-access-qsrgj") pod "4d8eef16-f595-41be-9e77-e61a0235d6d9" (UID: "4d8eef16-f595-41be-9e77-e61a0235d6d9"). InnerVolumeSpecName "kube-api-access-qsrgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:48:55.956180 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:55.956150 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsrgj\" (UniqueName: \"kubernetes.io/projected/4d8eef16-f595-41be-9e77-e61a0235d6d9-kube-api-access-qsrgj\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:48:55.956180 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:55.956176 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d8eef16-f595-41be-9e77-e61a0235d6d9-maas-api-tls\") on node \"ip-10-0-133-147.ec2.internal\" DevicePath \"\"" Jul 02 19:48:55.989581 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:55.989552 2562 generic.go:356] "Generic (PLEG): container finished" podID="4d8eef16-f595-41be-9e77-e61a0235d6d9" containerID="ed7fd346bb2a07c685cf5bbb6df4a6485eca5d778d87753d8bdee7ceef1e2c85" exitCode=0 Jul 02 19:48:55.989716 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:55.989620 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb" Jul 02 19:48:55.989716 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:55.989634 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb" event={"ID":"4d8eef16-f595-41be-9e77-e61a0235d6d9","Type":"ContainerDied","Data":"ed7fd346bb2a07c685cf5bbb6df4a6485eca5d778d87753d8bdee7ceef1e2c85"} Jul 02 19:48:55.989716 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:55.989685 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb" event={"ID":"4d8eef16-f595-41be-9e77-e61a0235d6d9","Type":"ContainerDied","Data":"d1076806b5296394e52835d99cc35b5f4d7660767acf73523b5ecf0cf4ef3d31"} Jul 02 19:48:55.989716 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:55.989710 2562 scope.go:117] "RemoveContainer" containerID="ed7fd346bb2a07c685cf5bbb6df4a6485eca5d778d87753d8bdee7ceef1e2c85" Jul 02 19:48:56.001629 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:56.001584 2562 scope.go:117] "RemoveContainer" containerID="ed7fd346bb2a07c685cf5bbb6df4a6485eca5d778d87753d8bdee7ceef1e2c85" Jul 02 19:48:56.001881 ip-10-0-133-147 kubenswrapper[2562]: E0702 19:48:56.001858 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed7fd346bb2a07c685cf5bbb6df4a6485eca5d778d87753d8bdee7ceef1e2c85\": container with ID starting with ed7fd346bb2a07c685cf5bbb6df4a6485eca5d778d87753d8bdee7ceef1e2c85 not found: ID does not exist" containerID="ed7fd346bb2a07c685cf5bbb6df4a6485eca5d778d87753d8bdee7ceef1e2c85" Jul 02 19:48:56.001978 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:56.001890 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed7fd346bb2a07c685cf5bbb6df4a6485eca5d778d87753d8bdee7ceef1e2c85"} err="failed to get container status \"ed7fd346bb2a07c685cf5bbb6df4a6485eca5d778d87753d8bdee7ceef1e2c85\": rpc error: code = NotFound desc = could not find container \"ed7fd346bb2a07c685cf5bbb6df4a6485eca5d778d87753d8bdee7ceef1e2c85\": container with ID starting with ed7fd346bb2a07c685cf5bbb6df4a6485eca5d778d87753d8bdee7ceef1e2c85 not found: ID does not exist" Jul 02 19:48:56.015879 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:56.015854 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb"] Jul 02 19:48:56.020694 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:56.020668 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-0c31dc-5bc464ffbb-9z9bb"] Jul 02 19:48:56.777236 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:48:56.777208 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d8eef16-f595-41be-9e77-e61a0235d6d9" path="/var/lib/kubelet/pods/4d8eef16-f595-41be-9e77-e61a0235d6d9/volumes" Jul 02 19:49:01.336941 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:01.336908 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:49:01.451324 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:01.451292 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:49:01.669988 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:01.669894 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:49:01.942114 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:01.942031 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:49:07.409667 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:07.409635 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-57b5579877-bg8h6_6d41ed33-5de8-4d2a-86bd-3815fc777516/maas-api/0.log" Jul 02 19:49:07.522585 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:07.522517 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29717010-427c6_6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903/cleanup/0.log" Jul 02 19:49:07.634489 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:07.634462 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29717025-qrrfl_3716bd2d-c17e-4613-ab71-c78b272606fa/cleanup/0.log" Jul 02 19:49:07.770265 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:07.770237 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c8858bc75-t4lq9_e639f3e4-a42f-4252-9779-62c70a052603/manager/0.log" Jul 02 19:49:08.271243 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:08.271219 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d8ddf44b5-scdsf_64fb8e77-6fa0-4bce-a802-d8375e44a592/copy-manifests/0.log" Jul 02 19:49:08.289996 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:08.289971 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d8ddf44b5-scdsf_64fb8e77-6fa0-4bce-a802-d8375e44a592/manager/0.log" Jul 02 19:49:08.529701 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:08.529627 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-bsbw7_bb335b6a-1323-40e5-b2d6-4a426c678ef2/postgres/0.log" Jul 02 19:49:11.204881 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:11.204848 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-hwrrz_97b0bd4d-3d86-4b01-b9ac-ad625173fac7/discovery/0.log" Jul 02 19:49:11.688794 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:11.688765 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-qp4rb_5c8a5510-9e86-44dc-98ab-600383f4b571/payload-pre-processing/0.log" Jul 02 19:49:11.814568 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:11.814530 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-889xr_748bb359-fd6a-4f3a-91e9-c2fe8e47d941/payload-processing/0.log" Jul 02 19:49:12.274080 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:12.274052 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-mg4gr_b3d9b000-85a2-4def-9a25-105a85023e15/main/0.log" Jul 02 19:49:12.501337 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:12.501308 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-48l9l_490379dc-0d79-4b3f-a1e4-35bb39dc4f5d/main/0.log" Jul 02 19:49:12.751212 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:12.751108 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-lkd7b_e457b0f4-4de5-47ee-8f0f-935b4d4c81e6/main/0.log" Jul 02 19:49:16.635451 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:16.635416 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:49:16.734474 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:16.734442 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" limit=1024 Jul 02 19:49:19.482026 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:19.481990 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hqszg_448316b6-cd2d-4c73-8ca6-103e4c64f18e/global-pull-secret-syncer/0.log" Jul 02 19:49:19.646796 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:19.646770 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-75ftq_270e3b43-4182-4af0-ba88-b327c157291e/konnectivity-agent/0.log" Jul 02 19:49:19.743705 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:19.743626 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-147.ec2.internal_d10855b2eb41ac40b2a760c19b9dbc20/haproxy/0.log" Jul 02 19:49:26.556738 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:26.556703 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d3d6abf7-f154-43b1-8980-f06175e47018/alertmanager/0.log" Jul 02 19:49:26.582313 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:26.582283 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d3d6abf7-f154-43b1-8980-f06175e47018/config-reloader/0.log" Jul 02 19:49:26.610189 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:26.610114 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d3d6abf7-f154-43b1-8980-f06175e47018/kube-rbac-proxy-web/0.log" Jul 02 19:49:26.644018 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:26.643992 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d3d6abf7-f154-43b1-8980-f06175e47018/kube-rbac-proxy/0.log" Jul 02 19:49:26.674534 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:26.674519 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d3d6abf7-f154-43b1-8980-f06175e47018/kube-rbac-proxy-metric/0.log" Jul 02 19:49:26.705396 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:26.705378 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d3d6abf7-f154-43b1-8980-f06175e47018/prom-label-proxy/0.log" Jul 02 19:49:26.728837 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:26.728813 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d3d6abf7-f154-43b1-8980-f06175e47018/init-config-reloader/0.log" Jul 02 19:49:26.808283 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:26.808257 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-jgh6j_a3338032-5243-4ed5-8ed8-6ad89c8b9a55/kube-state-metrics/0.log" Jul 02 19:49:26.829243 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:26.829221 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-jgh6j_a3338032-5243-4ed5-8ed8-6ad89c8b9a55/kube-rbac-proxy-main/0.log" Jul 02 19:49:26.853476 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:26.853452 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-jgh6j_a3338032-5243-4ed5-8ed8-6ad89c8b9a55/kube-rbac-proxy-self/0.log" Jul 02 19:49:26.883753 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:26.883693 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-555f6bb4cf-s5gfc_95367456-3d2d-465e-b1a9-59c6f8fc66a1/metrics-server/0.log" Jul 02 19:49:26.908953 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:26.908933 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-vbd2x_26588082-05d8-446e-ad43-96ce0e8e5d96/monitoring-plugin/0.log" Jul 02 19:49:27.022021 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.021994 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m8g2g_5bfe4789-0421-4ae4-b909-10670eadfecd/node-exporter/0.log" Jul 02 19:49:27.046174 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.046155 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m8g2g_5bfe4789-0421-4ae4-b909-10670eadfecd/kube-rbac-proxy/0.log" Jul 02 19:49:27.074217 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.074190 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m8g2g_5bfe4789-0421-4ae4-b909-10670eadfecd/init-textfile/0.log" Jul 02 19:49:27.210526 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.210449 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-rb49r_788118e6-7cf7-4c96-aa08-7bebc980f993/kube-rbac-proxy-main/0.log" Jul 02 19:49:27.270839 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.270816 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-rb49r_788118e6-7cf7-4c96-aa08-7bebc980f993/kube-rbac-proxy-self/0.log" Jul 02 19:49:27.338151 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.338127 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-rb49r_788118e6-7cf7-4c96-aa08-7bebc980f993/openshift-state-metrics/0.log" Jul 02 19:49:27.403733 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.403714 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ca978e38-b363-422c-8c68-ea06ad2b98c1/prometheus/0.log" Jul 02 19:49:27.429757 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.429737 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ca978e38-b363-422c-8c68-ea06ad2b98c1/config-reloader/0.log" Jul 02 19:49:27.461696 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.461642 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ca978e38-b363-422c-8c68-ea06ad2b98c1/thanos-sidecar/0.log" Jul 02 19:49:27.487167 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.487147 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ca978e38-b363-422c-8c68-ea06ad2b98c1/kube-rbac-proxy-web/0.log" Jul 02 19:49:27.511298 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.511271 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ca978e38-b363-422c-8c68-ea06ad2b98c1/kube-rbac-proxy/0.log" Jul 02 19:49:27.536762 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.536732 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ca978e38-b363-422c-8c68-ea06ad2b98c1/kube-rbac-proxy-thanos/0.log" Jul 02 19:49:27.563644 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.563619 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ca978e38-b363-422c-8c68-ea06ad2b98c1/init-config-reloader/0.log" Jul 02 19:49:27.600769 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.600746 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-rc8xt_8148a379-8c15-4b5c-9a32-ad0d081a7205/prometheus-operator/0.log" Jul 02 19:49:27.624122 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.624102 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-rc8xt_8148a379-8c15-4b5c-9a32-ad0d081a7205/kube-rbac-proxy/0.log" Jul 02 19:49:27.651215 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.651193 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-vrmvk_9c68493d-8279-4226-9970-d63f8dba9acb/prometheus-operator-admission-webhook/0.log" Jul 02 19:49:27.689178 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.689154 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-587fc79689-mqtmt_e86d2639-7d04-4eb8-8b5d-2b79101a066f/telemeter-client/0.log" Jul 02 19:49:27.711901 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.711840 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-587fc79689-mqtmt_e86d2639-7d04-4eb8-8b5d-2b79101a066f/reload/0.log" Jul 02 19:49:27.734263 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.734242 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-587fc79689-mqtmt_e86d2639-7d04-4eb8-8b5d-2b79101a066f/kube-rbac-proxy/0.log" Jul 02 19:49:27.778202 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.778174 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d645bff7-k9gt7_dedb7476-dbe0-4bde-89a6-391c39a4057e/thanos-query/0.log" Jul 02 19:49:27.802115 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.802093 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d645bff7-k9gt7_dedb7476-dbe0-4bde-89a6-391c39a4057e/kube-rbac-proxy-web/0.log" Jul 02 19:49:27.823293 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.823275 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d645bff7-k9gt7_dedb7476-dbe0-4bde-89a6-391c39a4057e/kube-rbac-proxy/0.log" Jul 02 19:49:27.849605 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.849569 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d645bff7-k9gt7_dedb7476-dbe0-4bde-89a6-391c39a4057e/prom-label-proxy/0.log" Jul 02 19:49:27.871676 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.871654 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d645bff7-k9gt7_dedb7476-dbe0-4bde-89a6-391c39a4057e/kube-rbac-proxy-rules/0.log" Jul 02 19:49:27.893365 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:27.893348 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9d645bff7-k9gt7_dedb7476-dbe0-4bde-89a6-391c39a4057e/kube-rbac-proxy-metrics/0.log" Jul 02 19:49:29.715097 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.715046 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8"] Jul 02 19:49:29.715505 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.715491 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d8eef16-f595-41be-9e77-e61a0235d6d9" containerName="maas-api" Jul 02 19:49:29.715550 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.715506 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d8eef16-f595-41be-9e77-e61a0235d6d9" containerName="maas-api" Jul 02 19:49:29.715550 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.715516 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3716bd2d-c17e-4613-ab71-c78b272606fa" containerName="cleanup" Jul 02 19:49:29.715550 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.715522 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3716bd2d-c17e-4613-ab71-c78b272606fa" containerName="cleanup" Jul 02 19:49:29.715550 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.715537 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9beb6e93-7821-4ade-91e4-adf74805fb87" containerName="maas-api" Jul 02 19:49:29.715550 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.715542 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9beb6e93-7821-4ade-91e4-adf74805fb87" containerName="maas-api" Jul 02 19:49:29.715723 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.715624 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d8eef16-f595-41be-9e77-e61a0235d6d9" containerName="maas-api" Jul 02 19:49:29.715723 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.715633 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3716bd2d-c17e-4613-ab71-c78b272606fa" containerName="cleanup" Jul 02 19:49:29.715723 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.715643 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9beb6e93-7821-4ade-91e4-adf74805fb87" containerName="maas-api" Jul 02 19:49:29.719140 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.719124 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8" Jul 02 19:49:29.721924 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.721894 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ll6bw\"/\"default-dockercfg-9shf6\"" Jul 02 19:49:29.721924 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.721894 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ll6bw\"/\"openshift-service-ca.crt\"" Jul 02 19:49:29.722118 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.721935 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ll6bw\"/\"kube-root-ca.crt\"" Jul 02 19:49:29.727688 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.727667 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8"] Jul 02 19:49:29.767902 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.767877 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fbf4b426-f521-48c2-8738-a24e8b26fcfb-sys\") pod \"perf-node-gather-daemonset-dstn8\" (UID: \"fbf4b426-f521-48c2-8738-a24e8b26fcfb\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8" Jul 02 19:49:29.768037 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.767921 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9p6w\" (UniqueName: \"kubernetes.io/projected/fbf4b426-f521-48c2-8738-a24e8b26fcfb-kube-api-access-x9p6w\") pod \"perf-node-gather-daemonset-dstn8\" (UID: \"fbf4b426-f521-48c2-8738-a24e8b26fcfb\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8" Jul 02 19:49:29.768037 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.768001 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fbf4b426-f521-48c2-8738-a24e8b26fcfb-podres\") pod \"perf-node-gather-daemonset-dstn8\" (UID: \"fbf4b426-f521-48c2-8738-a24e8b26fcfb\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8" Jul 02 19:49:29.768147 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.768045 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fbf4b426-f521-48c2-8738-a24e8b26fcfb-lib-modules\") pod \"perf-node-gather-daemonset-dstn8\" (UID: \"fbf4b426-f521-48c2-8738-a24e8b26fcfb\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8" Jul 02 19:49:29.768147 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.768082 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fbf4b426-f521-48c2-8738-a24e8b26fcfb-proc\") pod \"perf-node-gather-daemonset-dstn8\" (UID: \"fbf4b426-f521-48c2-8738-a24e8b26fcfb\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8" Jul 02 19:49:29.868889 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.868856 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fbf4b426-f521-48c2-8738-a24e8b26fcfb-sys\") pod \"perf-node-gather-daemonset-dstn8\" (UID: \"fbf4b426-f521-48c2-8738-a24e8b26fcfb\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8" Jul 02 19:49:29.869023 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.868897 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9p6w\" (UniqueName: \"kubernetes.io/projected/fbf4b426-f521-48c2-8738-a24e8b26fcfb-kube-api-access-x9p6w\") pod \"perf-node-gather-daemonset-dstn8\" (UID: \"fbf4b426-f521-48c2-8738-a24e8b26fcfb\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8" Jul 02 19:49:29.869023 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.868929 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fbf4b426-f521-48c2-8738-a24e8b26fcfb-podres\") pod \"perf-node-gather-daemonset-dstn8\" (UID: \"fbf4b426-f521-48c2-8738-a24e8b26fcfb\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8" Jul 02 19:49:29.869023 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.868958 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fbf4b426-f521-48c2-8738-a24e8b26fcfb-lib-modules\") pod \"perf-node-gather-daemonset-dstn8\" (UID: \"fbf4b426-f521-48c2-8738-a24e8b26fcfb\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8" Jul 02 19:49:29.869023 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.868994 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fbf4b426-f521-48c2-8738-a24e8b26fcfb-proc\") pod \"perf-node-gather-daemonset-dstn8\" (UID: \"fbf4b426-f521-48c2-8738-a24e8b26fcfb\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8" Jul 02 19:49:29.869023 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.868996 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fbf4b426-f521-48c2-8738-a24e8b26fcfb-sys\") pod \"perf-node-gather-daemonset-dstn8\" (UID: \"fbf4b426-f521-48c2-8738-a24e8b26fcfb\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8" Jul 02 19:49:29.869239 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.869113 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fbf4b426-f521-48c2-8738-a24e8b26fcfb-lib-modules\") pod \"perf-node-gather-daemonset-dstn8\" (UID: \"fbf4b426-f521-48c2-8738-a24e8b26fcfb\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8" Jul 02 19:49:29.869239 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.869125 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fbf4b426-f521-48c2-8738-a24e8b26fcfb-podres\") pod \"perf-node-gather-daemonset-dstn8\" (UID: \"fbf4b426-f521-48c2-8738-a24e8b26fcfb\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8" Jul 02 19:49:29.869239 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.869148 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fbf4b426-f521-48c2-8738-a24e8b26fcfb-proc\") pod \"perf-node-gather-daemonset-dstn8\" (UID: \"fbf4b426-f521-48c2-8738-a24e8b26fcfb\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8" Jul 02 19:49:29.877608 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:29.877577 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9p6w\" (UniqueName: \"kubernetes.io/projected/fbf4b426-f521-48c2-8738-a24e8b26fcfb-kube-api-access-x9p6w\") pod \"perf-node-gather-daemonset-dstn8\" (UID: \"fbf4b426-f521-48c2-8738-a24e8b26fcfb\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8" Jul 02 19:49:30.030644 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:30.030620 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8" Jul 02 19:49:30.165888 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:30.165863 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8"] Jul 02 19:49:30.167290 ip-10-0-133-147 kubenswrapper[2562]: W0702 19:49:30.167265 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfbf4b426_f521_48c2_8738_a24e8b26fcfb.slice/crio-fc193ace0c3f5aaaabb810a45702cd8c086b38ebb6e284c8817b305a3e6734a9 WatchSource:0}: Error finding container fc193ace0c3f5aaaabb810a45702cd8c086b38ebb6e284c8817b305a3e6734a9: Status 404 returned error can't find the container with id fc193ace0c3f5aaaabb810a45702cd8c086b38ebb6e284c8817b305a3e6734a9 Jul 02 19:49:30.366346 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:30.366268 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6756cc7865-5jlzt_a244ef32-fc8f-4eb5-9866-e9d588e8fb54/console/0.log" Jul 02 19:49:31.172680 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:31.172627 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8" event={"ID":"fbf4b426-f521-48c2-8738-a24e8b26fcfb","Type":"ContainerStarted","Data":"617e03b1799e3066bf2c45c39869b7820161b25a717b95b1ea2035353aa305d2"} Jul 02 19:49:31.173084 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:31.172698 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8" event={"ID":"fbf4b426-f521-48c2-8738-a24e8b26fcfb","Type":"ContainerStarted","Data":"fc193ace0c3f5aaaabb810a45702cd8c086b38ebb6e284c8817b305a3e6734a9"} Jul 02 19:49:31.173084 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:31.172804 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8" Jul 02 19:49:31.191670 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:31.191621 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8" podStartSLOduration=2.191606488 podStartE2EDuration="2.191606488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:49:31.190145036 +0000 UTC m=+2478.967162615" watchObservedRunningTime="2026-07-02 19:49:31.191606488 +0000 UTC m=+2478.968624103" Jul 02 19:49:32.139548 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:32.139521 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9pdxg_b070f3c1-e648-48ab-9af8-348a82ea5404/dns/0.log" Jul 02 19:49:32.160088 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:32.160066 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9pdxg_b070f3c1-e648-48ab-9af8-348a82ea5404/kube-rbac-proxy/0.log" Jul 02 19:49:32.182808 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:32.182783 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-45nlz_c07acb28-0571-41c0-8a02-55cac407c73d/dns-node-resolver/0.log" Jul 02 19:49:32.853139 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:32.853108 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xzqgm_51243d8b-5625-444e-bc70-fa68e9b13ecb/node-ca/0.log" Jul 02 19:49:34.033743 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:34.033699 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-hwrrz_97b0bd4d-3d86-4b01-b9ac-ad625173fac7/discovery/0.log" Jul 02 19:49:34.182818 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:34.182789 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-qp4rb_5c8a5510-9e86-44dc-98ab-600383f4b571/payload-pre-processing/0.log" Jul 02 19:49:34.259639 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:34.259613 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-889xr_748bb359-fd6a-4f3a-91e9-c2fe8e47d941/payload-processing/0.log" Jul 02 19:49:34.973075 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:34.973044 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2mrvq_3e079bf3-979e-4a34-9f19-8c6633bd0a15/serve-healthcheck-canary/0.log" Jul 02 19:49:35.762317 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:35.762278 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v7lc8_0499397a-c856-416b-b3e2-3b3998080ea7/kube-rbac-proxy/0.log" Jul 02 19:49:35.782089 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:35.782067 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v7lc8_0499397a-c856-416b-b3e2-3b3998080ea7/exporter/0.log" Jul 02 19:49:35.803508 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:35.803488 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-v7lc8_0499397a-c856-416b-b3e2-3b3998080ea7/extractor/0.log" Jul 02 19:49:37.189155 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:37.189127 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-dstn8" Jul 02 19:49:38.100513 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:38.100483 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-57b5579877-bg8h6_6d41ed33-5de8-4d2a-86bd-3815fc777516/maas-api/0.log" Jul 02 19:49:38.123815 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:38.123789 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29717010-427c6_6bb3ae2d-f06c-4cf7-a01f-f19ba38b5903/cleanup/0.log" Jul 02 19:49:38.150645 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:38.150617 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29717025-qrrfl_3716bd2d-c17e-4613-ab71-c78b272606fa/cleanup/0.log" Jul 02 19:49:38.311542 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:38.311511 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-c8858bc75-t4lq9_e639f3e4-a42f-4252-9779-62c70a052603/manager/0.log" Jul 02 19:49:38.571194 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:38.571165 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d8ddf44b5-scdsf_64fb8e77-6fa0-4bce-a802-d8375e44a592/manager/0.log" Jul 02 19:49:38.589370 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:38.589343 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d8ddf44b5-scdsf_64fb8e77-6fa0-4bce-a802-d8375e44a592/copy-manifests/0.log" Jul 02 19:49:38.682890 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:38.682863 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-bsbw7_bb335b6a-1323-40e5-b2d6-4a426c678ef2/postgres/0.log" Jul 02 19:49:40.170009 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:40.169933 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7d75bb4b7f-2vhgv_392123d7-fc1b-45ab-a795-197713c10261/manager/0.log" Jul 02 19:49:40.609261 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:40.609224 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-hwrrz_97b0bd4d-3d86-4b01-b9ac-ad625173fac7/discovery/0.log" Jul 02 19:49:40.735674 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:40.735632 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-qp4rb_5c8a5510-9e86-44dc-98ab-600383f4b571/payload-pre-processing/0.log" Jul 02 19:49:40.805869 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:40.805845 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-889xr_748bb359-fd6a-4f3a-91e9-c2fe8e47d941/payload-processing/0.log" Jul 02 19:49:42.282919 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:42.282876 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-hwrrz_97b0bd4d-3d86-4b01-b9ac-ad625173fac7/discovery/0.log" Jul 02 19:49:42.399332 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:42.399289 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-qp4rb_5c8a5510-9e86-44dc-98ab-600383f4b571/payload-pre-processing/0.log" Jul 02 19:49:42.471397 ip-10-0-133-147 kubenswrapper[2562]: I0702 19:49:42.471371 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-889xr_748bb359-fd6a-4f3a-91e9-c2fe8e47d941/payload-processing/0.log"