Jul 29 00:54:09.623099 ip-10-0-132-136 systemd[1]: Starting Kubernetes Kubelet... Jul 29 00:54:09.977330 ip-10-0-132-136 kubenswrapper[2543]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 00:54:09.977330 ip-10-0-132-136 kubenswrapper[2543]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 29 00:54:09.977330 ip-10-0-132-136 kubenswrapper[2543]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 00:54:09.977330 ip-10-0-132-136 kubenswrapper[2543]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 29 00:54:09.977330 ip-10-0-132-136 kubenswrapper[2543]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 00:54:09.978389 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.978306 2543 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 29 00:54:09.984132 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984117 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 00:54:09.984132 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984133 2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 00:54:09.984200 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984137 2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 00:54:09.984200 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984140 2543 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 00:54:09.984200 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984143 2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 00:54:09.984200 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984148 2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 00:54:09.984200 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984151 2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 00:54:09.984200 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984154 2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 00:54:09.984200 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984160 2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 00:54:09.984200 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984163 2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 00:54:09.984200 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984166 2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 00:54:09.984200 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984169 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 00:54:09.984200 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984172 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 00:54:09.984200 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984176 2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 00:54:09.984200 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984178 2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 00:54:09.984200 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984181 2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 00:54:09.984200 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984185 2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 00:54:09.984200 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984188 2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 00:54:09.984200 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984192 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 00:54:09.984200 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984195 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 00:54:09.984200 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984198 2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 00:54:09.984825 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984200 2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 00:54:09.984825 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984203 2543 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 00:54:09.984825 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984207 2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 00:54:09.984825 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984210 2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 00:54:09.984825 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984212 2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 00:54:09.984825 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984215 2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 00:54:09.984825 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984218 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 00:54:09.984825 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984221 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 00:54:09.984825 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984223 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 00:54:09.984825 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984225 2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 00:54:09.984825 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984228 2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 00:54:09.984825 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984230 2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 00:54:09.984825 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984234 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 00:54:09.984825 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984236 2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 00:54:09.984825 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984239 2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 00:54:09.984825 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984242 2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 00:54:09.984825 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984244 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 00:54:09.984825 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984247 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 00:54:09.984825 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984249 2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 00:54:09.984825 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984252 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 00:54:09.985385 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984255 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 00:54:09.985385 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984257 2543 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 00:54:09.985385 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984260 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 00:54:09.985385 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984262 2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 00:54:09.985385 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984265 2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 00:54:09.985385 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984267 2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 00:54:09.985385 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984269 2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 00:54:09.985385 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984272 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 00:54:09.985385 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984274 2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 00:54:09.985385 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984277 2543 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 00:54:09.985385 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984279 2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 00:54:09.985385 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984282 2543 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 00:54:09.985385 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984284 2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 00:54:09.985385 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984288 2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 00:54:09.985385 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984290 2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 00:54:09.985385 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984294 2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 00:54:09.985385 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984298 2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 00:54:09.985385 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984301 2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 00:54:09.985385 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984304 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 00:54:09.985385 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984307 2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 00:54:09.986089 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984310 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 00:54:09.986089 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984313 2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 00:54:09.986089 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984315 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 00:54:09.986089 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984317 2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 00:54:09.986089 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984320 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 00:54:09.986089 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984323 2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 00:54:09.986089 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984325 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 00:54:09.986089 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984328 2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 00:54:09.986089 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984331 2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 00:54:09.986089 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984333 2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 00:54:09.986089 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984336 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 00:54:09.986089 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984339 2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 00:54:09.986089 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984341 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 00:54:09.986089 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984344 2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 00:54:09.986089 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984347 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 00:54:09.986089 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984349 2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 00:54:09.986089 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984352 2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 00:54:09.986089 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984354 2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 00:54:09.986089 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984356 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 00:54:09.986089 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984359 2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 00:54:09.986589 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984362 2543 feature_gate.go:328] unrecognized feature gate: Example Jul 29 00:54:09.986589 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984364 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 00:54:09.986589 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984367 2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 00:54:09.986589 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984370 2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 00:54:09.986589 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984373 2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 00:54:09.986589 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984376 2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 00:54:09.986589 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984378 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 00:54:09.986589 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984381 2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 00:54:09.986589 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984383 2543 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 00:54:09.986589 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984386 2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 00:54:09.986589 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984388 2543 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 00:54:09.986589 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984391 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 00:54:09.986589 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984393 2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 00:54:09.986589 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984396 2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 00:54:09.986589 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984398 2543 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 00:54:09.986589 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984401 2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 00:54:09.986589 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984404 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 00:54:09.986589 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.984406 2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 00:54:09.987048 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986795 2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 00:54:09.987048 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986805 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 00:54:09.987048 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986809 2543 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 00:54:09.987048 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986833 2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 00:54:09.987048 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986838 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 00:54:09.987048 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986841 2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 00:54:09.987048 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986845 2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 00:54:09.987048 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986848 2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 00:54:09.987048 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986851 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 00:54:09.987048 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986854 2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 00:54:09.987048 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986857 2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 00:54:09.987048 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986860 2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 00:54:09.987048 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986863 2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 00:54:09.987048 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986866 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 00:54:09.987048 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986869 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 00:54:09.987048 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986872 2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 00:54:09.987048 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986875 2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 00:54:09.987048 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986880 2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 00:54:09.987048 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986884 2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 00:54:09.987525 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986888 2543 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 00:54:09.987525 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986891 2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 00:54:09.987525 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986894 2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 00:54:09.987525 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986897 2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 00:54:09.987525 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986900 2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 00:54:09.987525 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986903 2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 00:54:09.987525 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986905 2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 00:54:09.987525 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986908 2543 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 00:54:09.987525 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986911 2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 00:54:09.987525 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986914 2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 00:54:09.987525 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986917 2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 00:54:09.987525 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986920 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 00:54:09.987525 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986923 2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 00:54:09.987525 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986927 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 00:54:09.987525 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986931 2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 00:54:09.987525 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986934 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 00:54:09.987525 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986937 2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 00:54:09.987525 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986939 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 00:54:09.987525 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986942 2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 00:54:09.987525 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986945 2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 00:54:09.988036 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986947 2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 00:54:09.988036 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986950 2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 00:54:09.988036 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986952 2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 00:54:09.988036 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986955 2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 00:54:09.988036 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986957 2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 00:54:09.988036 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986960 2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 00:54:09.988036 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986962 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 00:54:09.988036 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986965 2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 00:54:09.988036 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986967 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 00:54:09.988036 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986970 2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 00:54:09.988036 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986973 2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 00:54:09.988036 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986976 2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 00:54:09.988036 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986979 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 00:54:09.988036 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986981 2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 00:54:09.988036 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986984 2543 feature_gate.go:328] unrecognized feature gate: Example Jul 29 00:54:09.988036 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986986 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 00:54:09.988036 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986989 2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 00:54:09.988036 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986991 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 00:54:09.988036 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986995 2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 00:54:09.988036 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.986997 2543 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 00:54:09.988507 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987000 2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 00:54:09.988507 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987003 2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 00:54:09.988507 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987005 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 00:54:09.988507 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987008 2543 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 00:54:09.988507 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987010 2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 00:54:09.988507 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987013 2543 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 00:54:09.988507 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987018 2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 00:54:09.988507 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987021 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 00:54:09.988507 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987024 2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 00:54:09.988507 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987027 2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 00:54:09.988507 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987030 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 00:54:09.988507 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987033 2543 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 00:54:09.988507 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987035 2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 00:54:09.988507 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987038 2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 00:54:09.988507 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987040 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 00:54:09.988507 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987043 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 00:54:09.988507 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987045 2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 00:54:09.988507 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987048 2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 00:54:09.988507 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987050 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 00:54:09.988992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987053 2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 00:54:09.988992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987056 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 00:54:09.988992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987058 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 00:54:09.988992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987061 2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 00:54:09.988992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987063 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 00:54:09.988992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987066 2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 00:54:09.988992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987069 2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 00:54:09.988992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987071 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 00:54:09.988992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987074 2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 00:54:09.988992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987076 2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 00:54:09.988992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987079 2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 00:54:09.988992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987081 2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 00:54:09.988992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987085 2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 00:54:09.988992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987088 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 00:54:09.988992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987091 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 00:54:09.988992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987093 2543 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 00:54:09.988992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987096 2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 00:54:09.988992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987098 2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 00:54:09.988992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987101 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 00:54:09.989460 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987104 2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 00:54:09.989460 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987107 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 00:54:09.989460 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987470 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 00:54:09.989460 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987475 2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 00:54:09.989460 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987478 2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 00:54:09.989460 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987480 2543 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 00:54:09.989460 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987483 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 00:54:09.989460 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987486 2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 00:54:09.989460 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987489 2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 00:54:09.989460 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987491 2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 00:54:09.989460 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987494 2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 00:54:09.989460 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987496 2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 00:54:09.989460 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987499 2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 00:54:09.989460 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987501 2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 00:54:09.989460 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987504 2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 00:54:09.989460 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987507 2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 00:54:09.989460 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987509 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 00:54:09.989460 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987512 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 00:54:09.989460 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987515 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 00:54:09.989460 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987517 2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 00:54:09.989960 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987520 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 00:54:09.989960 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987522 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 00:54:09.989960 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987525 2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 00:54:09.989960 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987527 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 00:54:09.989960 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987530 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 00:54:09.989960 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987533 2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 00:54:09.989960 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987536 2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 00:54:09.989960 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987538 2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 00:54:09.989960 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987541 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 00:54:09.989960 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987544 2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 00:54:09.989960 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987546 2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 00:54:09.989960 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987549 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 00:54:09.989960 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987551 2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 00:54:09.989960 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987555 2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 00:54:09.989960 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987557 2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 00:54:09.989960 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987560 2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 00:54:09.989960 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987562 2543 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 00:54:09.989960 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987574 2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 00:54:09.989960 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987577 2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 00:54:09.989960 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987580 2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 00:54:09.990580 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987583 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 00:54:09.990580 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987586 2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 00:54:09.990580 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987589 2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 00:54:09.990580 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987592 2543 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 00:54:09.990580 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987594 2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 00:54:09.990580 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987597 2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 00:54:09.990580 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987599 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 00:54:09.990580 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987602 2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 00:54:09.990580 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987604 2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 00:54:09.990580 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987607 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 00:54:09.990580 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987610 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 00:54:09.990580 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987612 2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 00:54:09.990580 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987615 2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 00:54:09.990580 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987618 2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 00:54:09.990580 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987620 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 00:54:09.990580 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987622 2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 00:54:09.990580 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987625 2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 00:54:09.990580 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987628 2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 00:54:09.990580 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987631 2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 00:54:09.991059 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987635 2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 00:54:09.991059 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987638 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 00:54:09.991059 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987641 2543 feature_gate.go:328] unrecognized feature gate: Example Jul 29 00:54:09.991059 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987644 2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 00:54:09.991059 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987647 2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 00:54:09.991059 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987650 2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 00:54:09.991059 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987653 2543 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 00:54:09.991059 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987656 2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 00:54:09.991059 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987658 2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 00:54:09.991059 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987661 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 00:54:09.991059 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987664 2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 00:54:09.991059 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987666 2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 00:54:09.991059 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987669 2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 00:54:09.991059 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987671 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 00:54:09.991059 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987674 2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 00:54:09.991059 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987676 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 00:54:09.991059 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987679 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 00:54:09.991059 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987681 2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 00:54:09.991059 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987684 2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 00:54:09.991517 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987686 2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 00:54:09.991517 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987688 2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 00:54:09.991517 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987691 2543 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 00:54:09.991517 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987694 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 00:54:09.991517 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987696 2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 00:54:09.991517 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987699 2543 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 00:54:09.991517 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987702 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 00:54:09.991517 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987705 2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 00:54:09.991517 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987707 2543 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 00:54:09.991517 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987710 2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 00:54:09.991517 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987712 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 00:54:09.991517 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987714 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 00:54:09.991517 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987718 2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 00:54:09.991517 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987720 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 00:54:09.991517 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987723 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 00:54:09.991517 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987725 2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 00:54:09.991517 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987728 2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 00:54:09.991517 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987731 2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 00:54:09.991517 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987733 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987736 2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987740 2543 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987743 2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.987745 2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987828 2543 flags.go:64] FLAG: --address="0.0.0.0" Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987851 2543 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987857 2543 flags.go:64] FLAG: --anonymous-auth="true" Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987861 2543 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987865 2543 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987869 2543 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987873 2543 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987877 2543 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987880 2543 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987884 2543 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987887 2543 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987890 2543 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987893 2543 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987896 2543 flags.go:64] FLAG: --cgroup-root="" Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987899 2543 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987902 2543 flags.go:64] FLAG: --client-ca-file="" Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987906 2543 flags.go:64] FLAG: --cloud-provider="external" Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987909 2543 flags.go:64] FLAG: --cluster-dns="[]" Jul 29 00:54:09.991979 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987913 2543 flags.go:64] FLAG: --cluster-domain="" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987915 2543 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987919 2543 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987922 2543 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987925 2543 flags.go:64] FLAG: --container-log-max-files="5" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987929 2543 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987932 2543 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987935 2543 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987938 2543 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987941 2543 flags.go:64] FLAG: --contention-profiling="false" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987944 2543 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987947 2543 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987950 2543 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987953 2543 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987957 2543 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987960 2543 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987963 2543 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987966 2543 flags.go:64] FLAG: --enable-load-reader="false" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987969 2543 flags.go:64] FLAG: --enable-server="true" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987971 2543 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987975 2543 flags.go:64] FLAG: --event-burst="100" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987978 2543 flags.go:64] FLAG: --event-qps="50" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987981 2543 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987984 2543 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 29 00:54:09.992540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987987 2543 flags.go:64] FLAG: --eviction-hard="" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987991 2543 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987994 2543 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987996 2543 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.987999 2543 flags.go:64] FLAG: --eviction-soft="" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988002 2543 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988008 2543 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988012 2543 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988014 2543 flags.go:64] FLAG: --experimental-mounter-path="" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988017 2543 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988020 2543 flags.go:64] FLAG: --fail-swap-on="true" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988023 2543 flags.go:64] FLAG: --feature-gates="" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988026 2543 flags.go:64] FLAG: --file-check-frequency="20s" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988029 2543 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988032 2543 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988036 2543 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988039 2543 flags.go:64] FLAG: --healthz-port="10248" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988042 2543 flags.go:64] FLAG: --help="false" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988045 2543 flags.go:64] FLAG: --hostname-override="ip-10-0-132-136.ec2.internal" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988048 2543 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988051 2543 flags.go:64] FLAG: --http-check-frequency="20s" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988054 2543 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988058 2543 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 29 00:54:09.993135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988061 2543 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988064 2543 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988066 2543 flags.go:64] FLAG: --image-service-endpoint="" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988069 2543 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988072 2543 flags.go:64] FLAG: --kube-api-burst="100" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988077 2543 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988080 2543 flags.go:64] FLAG: --kube-api-qps="50" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988083 2543 flags.go:64] FLAG: --kube-reserved="" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988085 2543 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988088 2543 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988091 2543 flags.go:64] FLAG: --kubelet-cgroups="" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988094 2543 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988097 2543 flags.go:64] FLAG: --lock-file="" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988099 2543 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988102 2543 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988106 2543 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988111 2543 flags.go:64] FLAG: --log-json-split-stream="false" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988115 2543 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988118 2543 flags.go:64] FLAG: --log-text-split-stream="false" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988120 2543 flags.go:64] FLAG: --logging-format="text" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988123 2543 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988127 2543 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988129 2543 flags.go:64] FLAG: --manifest-url="" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988132 2543 flags.go:64] FLAG: --manifest-url-header="" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988136 2543 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 29 00:54:09.993711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988139 2543 flags.go:64] FLAG: --max-open-files="1000000" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988143 2543 flags.go:64] FLAG: --max-pods="110" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988146 2543 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988149 2543 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988152 2543 flags.go:64] FLAG: --memory-manager-policy="None" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988155 2543 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988158 2543 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988160 2543 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988163 2543 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988170 2543 flags.go:64] FLAG: --node-status-max-images="50" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988173 2543 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988176 2543 flags.go:64] FLAG: --oom-score-adj="-999" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988180 2543 flags.go:64] FLAG: --pod-cidr="" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988183 2543 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988188 2543 flags.go:64] FLAG: --pod-manifest-path="" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988191 2543 flags.go:64] FLAG: --pod-max-pids="-1" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988194 2543 flags.go:64] FLAG: --pods-per-core="0" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988197 2543 flags.go:64] FLAG: --port="10250" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988200 2543 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988202 2543 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0acec5e78e626858d" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988206 2543 flags.go:64] FLAG: --qos-reserved="" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988208 2543 flags.go:64] FLAG: --read-only-port="10255" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988211 2543 flags.go:64] FLAG: --register-node="true" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988215 2543 flags.go:64] FLAG: --register-with-taints="" Jul 29 00:54:09.994431 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988218 2543 flags.go:64] FLAG: --registry-burst="10" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988222 2543 flags.go:64] FLAG: --registry-qps="5" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988224 2543 flags.go:64] FLAG: --reserved-cpus="" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988227 2543 flags.go:64] FLAG: --reserved-memory="" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988231 2543 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988233 2543 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988236 2543 flags.go:64] FLAG: --rotate-certificates="false" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988239 2543 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988242 2543 flags.go:64] FLAG: --runonce="false" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988245 2543 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988247 2543 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988250 2543 flags.go:64] FLAG: --seccomp-default="false" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988253 2543 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988256 2543 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988259 2543 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988261 2543 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988264 2543 flags.go:64] FLAG: --storage-driver-password="root" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988267 2543 flags.go:64] FLAG: --storage-driver-secure="false" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988270 2543 flags.go:64] FLAG: --storage-driver-table="stats" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988273 2543 flags.go:64] FLAG: --storage-driver-user="root" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988277 2543 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988280 2543 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988283 2543 flags.go:64] FLAG: --system-cgroups="" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988285 2543 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988290 2543 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 29 00:54:09.995028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988293 2543 flags.go:64] FLAG: --tls-cert-file="" Jul 29 00:54:09.995638 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988296 2543 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 29 00:54:09.995638 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988300 2543 flags.go:64] FLAG: --tls-min-version="" Jul 29 00:54:09.995638 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988303 2543 flags.go:64] FLAG: --tls-private-key-file="" Jul 29 00:54:09.995638 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988306 2543 flags.go:64] FLAG: --topology-manager-policy="none" Jul 29 00:54:09.995638 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988308 2543 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 29 00:54:09.995638 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988312 2543 flags.go:64] FLAG: --topology-manager-scope="container" Jul 29 00:54:09.995638 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988315 2543 flags.go:64] FLAG: --v="2" Jul 29 00:54:09.995638 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988319 2543 flags.go:64] FLAG: --version="false" Jul 29 00:54:09.995638 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988324 2543 flags.go:64] FLAG: --vmodule="" Jul 29 00:54:09.995638 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988327 2543 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 29 00:54:09.995638 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.988330 2543 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 29 00:54:09.995638 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988437 2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 00:54:09.995638 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988440 2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 00:54:09.995638 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988444 2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 00:54:09.995638 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988447 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 00:54:09.995638 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988451 2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 00:54:09.995638 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988455 2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 00:54:09.995638 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988458 2543 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 00:54:09.995638 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988461 2543 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 00:54:09.995638 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988464 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 00:54:09.995638 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988466 2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 00:54:09.995638 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988469 2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 00:54:09.996193 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988472 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 00:54:09.996193 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988475 2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 00:54:09.996193 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988477 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 00:54:09.996193 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988480 2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 00:54:09.996193 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988484 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 00:54:09.996193 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988487 2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 00:54:09.996193 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988489 2543 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 00:54:09.996193 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988492 2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 00:54:09.996193 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988494 2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 00:54:09.996193 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988497 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 00:54:09.996193 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988499 2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 00:54:09.996193 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988502 2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 00:54:09.996193 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988504 2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 00:54:09.996193 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988507 2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 00:54:09.996193 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988509 2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 00:54:09.996193 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988513 2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 00:54:09.996193 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988516 2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 00:54:09.996193 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988519 2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 00:54:09.996193 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988521 2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 00:54:09.996193 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988525 2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 00:54:09.996684 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988527 2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 00:54:09.996684 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988530 2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 00:54:09.996684 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988532 2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 00:54:09.996684 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988535 2543 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 00:54:09.996684 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988538 2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 00:54:09.996684 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988540 2543 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 00:54:09.996684 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988543 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 00:54:09.996684 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988546 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 00:54:09.996684 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988548 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 00:54:09.996684 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988551 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 00:54:09.996684 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988553 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 00:54:09.996684 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988556 2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 00:54:09.996684 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988558 2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 00:54:09.996684 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988561 2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 00:54:09.996684 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988563 2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 00:54:09.996684 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988565 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 00:54:09.996684 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988569 2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 00:54:09.996684 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988572 2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 00:54:09.996684 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988574 2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 00:54:09.996684 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988577 2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 00:54:09.997177 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988579 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 00:54:09.997177 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988582 2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 00:54:09.997177 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988584 2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 00:54:09.997177 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988587 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 00:54:09.997177 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988589 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 00:54:09.997177 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988591 2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 00:54:09.997177 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988594 2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 00:54:09.997177 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988597 2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 00:54:09.997177 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988600 2543 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 00:54:09.997177 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988603 2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 00:54:09.997177 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988605 2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 00:54:09.997177 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988607 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 00:54:09.997177 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988611 2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 00:54:09.997177 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988613 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 00:54:09.997177 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988616 2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 00:54:09.997177 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988618 2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 00:54:09.997177 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988622 2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 00:54:09.997177 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988626 2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 00:54:09.997177 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988629 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 00:54:09.997177 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988632 2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 00:54:09.997685 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988634 2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 00:54:09.997685 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988637 2543 feature_gate.go:328] unrecognized feature gate: Example Jul 29 00:54:09.997685 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988639 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 00:54:09.997685 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988642 2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 00:54:09.997685 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988644 2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 00:54:09.997685 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988647 2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 00:54:09.997685 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988650 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 00:54:09.997685 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988652 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 00:54:09.997685 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988656 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 00:54:09.997685 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988659 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 00:54:09.997685 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988661 2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 00:54:09.997685 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988664 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 00:54:09.997685 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988666 2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 00:54:09.997685 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988669 2543 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 00:54:09.997685 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988671 2543 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 00:54:09.997685 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988674 2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 00:54:09.997685 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988676 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 00:54:09.997685 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988679 2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 00:54:09.997685 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988682 2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 00:54:09.997685 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988685 2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 00:54:09.998256 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988689 2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 00:54:09.998256 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988692 2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 00:54:09.998256 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988695 2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 00:54:09.998256 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988698 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 00:54:09.998256 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988700 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 00:54:09.998256 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988704 2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 00:54:09.998256 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988706 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 00:54:09.998256 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.988709 2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 00:54:09.998256 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.989381 2543 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 00:54:09.998256 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.995617 2543 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 29 00:54:09.998256 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.995633 2543 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 29 00:54:09.998256 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.996345 2543 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 29 00:54:09.998256 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.996356 2543 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 29 00:54:09.998256 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996413 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 00:54:09.998600 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996418 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 00:54:09.998600 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996422 2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 00:54:09.998600 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996425 2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 00:54:09.998600 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996428 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 00:54:09.998600 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996431 2543 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 00:54:09.998600 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996434 2543 feature_gate.go:328] unrecognized feature gate: Example Jul 29 00:54:09.998600 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996437 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 00:54:09.998600 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996440 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 00:54:09.998600 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996443 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 00:54:09.998600 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996446 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 00:54:09.998600 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996448 2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 00:54:09.998600 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996451 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 00:54:09.998600 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996454 2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 00:54:09.998600 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996457 2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 00:54:09.998600 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996459 2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 00:54:09.998600 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996462 2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 00:54:09.998600 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996465 2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 00:54:09.998600 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996467 2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 00:54:09.998600 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996470 2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 00:54:09.998600 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996473 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 00:54:09.999097 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996476 2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 00:54:09.999097 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996479 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 00:54:09.999097 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996481 2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 00:54:09.999097 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996485 2543 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 00:54:09.999097 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996488 2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 00:54:09.999097 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996490 2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 00:54:09.999097 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996493 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 00:54:09.999097 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996496 2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 00:54:09.999097 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996499 2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 00:54:09.999097 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996502 2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 00:54:09.999097 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996504 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 00:54:09.999097 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996507 2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 00:54:09.999097 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996510 2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 00:54:09.999097 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996512 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 00:54:09.999097 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996515 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 00:54:09.999097 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996517 2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 00:54:09.999097 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996520 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 00:54:09.999097 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996522 2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 00:54:09.999097 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996526 2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 00:54:09.999097 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996530 2543 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 00:54:09.999619 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996533 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 00:54:09.999619 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996536 2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 00:54:09.999619 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996539 2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 00:54:09.999619 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996541 2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 00:54:09.999619 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996544 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 00:54:09.999619 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996547 2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 00:54:09.999619 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996550 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 00:54:09.999619 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996552 2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 00:54:09.999619 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996555 2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 00:54:09.999619 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996557 2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 00:54:09.999619 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996560 2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 00:54:09.999619 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996562 2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 00:54:09.999619 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996565 2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 00:54:09.999619 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996567 2543 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 00:54:09.999619 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996570 2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 00:54:09.999619 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996572 2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 00:54:09.999619 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996576 2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 00:54:09.999619 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996579 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 00:54:09.999619 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996581 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 00:54:10.000131 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996584 2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 00:54:10.000131 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996588 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 00:54:10.000131 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996592 2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 00:54:10.000131 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996595 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 00:54:10.000131 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996599 2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 00:54:10.000131 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996602 2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 00:54:10.000131 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996604 2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 00:54:10.000131 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996607 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 00:54:10.000131 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996610 2543 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 00:54:10.000131 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996612 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 00:54:10.000131 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996615 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 00:54:10.000131 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996618 2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 00:54:10.000131 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996620 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 00:54:10.000131 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996623 2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 00:54:10.000131 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996627 2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 00:54:10.000131 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996631 2543 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 00:54:10.000131 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996633 2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 00:54:10.000131 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996636 2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 00:54:10.000131 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996639 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 00:54:10.000606 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996641 2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 00:54:10.000606 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996644 2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 00:54:10.000606 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996647 2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 00:54:10.000606 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996649 2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 00:54:10.000606 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996652 2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 00:54:10.000606 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996654 2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 00:54:10.000606 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996657 2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 00:54:10.000606 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996660 2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 00:54:10.000606 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996662 2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 00:54:10.000606 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996665 2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 00:54:10.000606 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996668 2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 00:54:10.000606 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996672 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 00:54:10.000606 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996674 2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 00:54:10.000606 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996677 2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 00:54:10.000606 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996680 2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 00:54:10.000606 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996683 2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 00:54:10.000606 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996685 2543 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 00:54:10.000606 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996689 2543 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 00:54:10.000606 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996691 2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 00:54:10.000606 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996694 2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 00:54:10.001106 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.996700 2543 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 00:54:10.001106 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996832 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 00:54:10.001106 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996836 2543 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 00:54:10.001106 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996839 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 00:54:10.001106 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996843 2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 00:54:10.001106 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996846 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 00:54:10.001106 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996850 2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 00:54:10.001106 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996853 2543 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 00:54:10.001106 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996855 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 00:54:10.001106 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996858 2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 00:54:10.001106 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996861 2543 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 00:54:10.001106 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996863 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 00:54:10.001106 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996866 2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 00:54:10.001106 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996868 2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 00:54:10.001106 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996871 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 00:54:10.001485 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996873 2543 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 00:54:10.001485 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996876 2543 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 00:54:10.001485 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996878 2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 00:54:10.001485 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996881 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 00:54:10.001485 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996883 2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 00:54:10.001485 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996886 2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 00:54:10.001485 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996888 2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 00:54:10.001485 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996891 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 00:54:10.001485 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996893 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 00:54:10.001485 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996897 2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 00:54:10.001485 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996899 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 00:54:10.001485 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996902 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 00:54:10.001485 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996905 2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 00:54:10.001485 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996907 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 00:54:10.001485 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996910 2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 00:54:10.001485 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996913 2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 00:54:10.001485 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996915 2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 00:54:10.001485 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996918 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 00:54:10.001485 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996920 2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 00:54:10.001485 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996923 2543 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 00:54:10.001992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996926 2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 00:54:10.001992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996928 2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 00:54:10.001992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996930 2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 00:54:10.001992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996933 2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 00:54:10.001992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996936 2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 00:54:10.001992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996938 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 00:54:10.001992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996941 2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 00:54:10.001992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996943 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 00:54:10.001992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996946 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 00:54:10.001992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996948 2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 00:54:10.001992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996951 2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 00:54:10.001992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996953 2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 00:54:10.001992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996955 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 00:54:10.001992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996958 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 00:54:10.001992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996960 2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 00:54:10.001992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996963 2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 00:54:10.001992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996965 2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 00:54:10.001992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996968 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 00:54:10.001992 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996972 2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 00:54:10.002482 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996975 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 00:54:10.002482 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996978 2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 00:54:10.002482 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996981 2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 00:54:10.002482 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996984 2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 00:54:10.002482 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996987 2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 00:54:10.002482 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996990 2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 00:54:10.002482 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996993 2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 00:54:10.002482 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996995 2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 00:54:10.002482 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.996998 2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 00:54:10.002482 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997000 2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 00:54:10.002482 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997003 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 00:54:10.002482 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997005 2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 00:54:10.002482 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997008 2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 00:54:10.002482 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997010 2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 00:54:10.002482 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997013 2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 00:54:10.002482 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997016 2543 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 00:54:10.002482 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997018 2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 00:54:10.002482 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997021 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 00:54:10.002482 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997024 2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 00:54:10.002482 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997028 2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 00:54:10.003074 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997031 2543 feature_gate.go:328] unrecognized feature gate: Example Jul 29 00:54:10.003074 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997034 2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 00:54:10.003074 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997037 2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 00:54:10.003074 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997039 2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 00:54:10.003074 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997042 2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 00:54:10.003074 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997045 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 00:54:10.003074 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997047 2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 00:54:10.003074 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997050 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 00:54:10.003074 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997052 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 00:54:10.003074 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997055 2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 00:54:10.003074 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997057 2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 00:54:10.003074 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997060 2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 00:54:10.003074 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997062 2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 00:54:10.003074 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997065 2543 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 00:54:10.003074 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997067 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 00:54:10.003074 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997070 2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 00:54:10.003074 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997074 2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 00:54:10.003074 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997077 2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 00:54:10.003074 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997080 2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 00:54:10.003074 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997083 2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 00:54:10.003584 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997085 2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 00:54:10.003584 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997087 2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 00:54:10.003584 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997090 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 00:54:10.003584 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997092 2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 00:54:10.003584 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997095 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 00:54:10.003584 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:09.997097 2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 00:54:10.003584 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.997102 2543 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 00:54:10.003584 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.997203 2543 server.go:962] "Client rotation is on, will bootstrap in background" Jul 29 00:54:10.003584 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:09.999226 2543 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 29 00:54:10.003584 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.000423 2543 server.go:1019] "Starting client certificate rotation" Jul 29 00:54:10.003584 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.000521 2543 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 00:54:10.003584 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.000559 2543 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 00:54:10.003584 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.002531 2543 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 00:54:10.004405 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.004390 2543 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 00:54:10.015016 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.014994 2543 log.go:25] "Validated CRI v1 runtime API" Jul 29 00:54:10.018307 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.018290 2543 log.go:25] "Validated CRI v1 image API" Jul 29 00:54:10.019138 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.019122 2543 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 29 00:54:10.022553 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.022534 2543 fs.go:135] Filesystem UUIDs: map[05d18c34-7f0b-436a-b413-23ca8f000668:/dev/nvme0n1p3 772704c9-5082-401a-b7d1-9a0f994fab04:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 29 00:54:10.022602 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.022553 2543 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 29 00:54:10.027729 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.027620 2543 manager.go:216] Machine: {Timestamp:2026-07-29 00:54:10.026558531 +0000 UTC m=+0.297202829 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095370 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec202fb21c02c39736b6c6ab4f9e5ec5 SystemUUID:ec202fb2-1c02-c397-36b6-c6ab4f9e5ec5 BootID:201e74f6-cfd2-4621-bc4d-d9474cf20354 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e3:37:d8:49:d5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e3:37:d8:49:d5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:d3:a3:bc:1a:f8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 29 00:54:10.027729 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.027726 2543 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 29 00:54:10.027841 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.027794 2543 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 29 00:54:10.027941 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.027926 2543 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 00:54:10.028676 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.028651 2543 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 29 00:54:10.028803 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.028680 2543 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-136.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 29 00:54:10.029364 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.029354 2543 topology_manager.go:138] "Creating topology manager with none policy" Jul 29 00:54:10.029396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.029366 2543 container_manager_linux.go:309] "Creating device plugin manager" Jul 29 00:54:10.029396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.029373 2543 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 00:54:10.029933 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.029922 2543 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 00:54:10.030713 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.030704 2543 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 29 00:54:10.032479 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.032468 2543 state_mem.go:36] "Initialized new in-memory state store" Jul 29 00:54:10.032596 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.032587 2543 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 29 00:54:10.032634 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.032626 2543 kubelet.go:486] "Attempting to sync node with API server" Jul 29 00:54:10.032662 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.032645 2543 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 29 00:54:10.032662 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.032659 2543 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 29 00:54:10.032711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.032668 2543 kubelet.go:398] "Adding apiserver pod source" Jul 29 00:54:10.032711 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.032679 2543 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 29 00:54:10.033045 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.033033 2543 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 00:54:10.033084 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.033057 2543 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 00:54:10.035996 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.035968 2543 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 29 00:54:10.037752 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.037738 2543 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 29 00:54:10.037827 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.037768 2543 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 29 00:54:10.039265 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.039250 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 29 00:54:10.039265 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.039266 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 29 00:54:10.039368 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.039272 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 29 00:54:10.039368 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.039277 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 29 00:54:10.039368 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.039286 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 29 00:54:10.039368 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.039292 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 29 00:54:10.039368 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.039300 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 29 00:54:10.039368 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.039304 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 29 00:54:10.039368 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.039310 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 29 00:54:10.039368 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.039315 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 29 00:54:10.039368 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.039329 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 29 00:54:10.039368 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.039334 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 29 00:54:10.040037 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.040026 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 29 00:54:10.040037 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.040037 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 29 00:54:10.040316 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.040284 2543 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 29 00:54:10.040627 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.040595 2543 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 29 00:54:10.042407 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.042375 2543 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-136.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 29 00:54:10.043327 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.043315 2543 server.go:1268] "Started kubelet" Jul 29 00:54:10.043442 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.043408 2543 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 29 00:54:10.043506 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.043408 2543 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 29 00:54:10.043506 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.043477 2543 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 29 00:54:10.044188 ip-10-0-132-136 systemd[1]: Started Kubernetes Kubelet. Jul 29 00:54:10.045341 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.045322 2543 server.go:310] "Adding debug handlers to kubelet server" Jul 29 00:54:10.045579 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.045562 2543 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 29 00:54:10.049214 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.049192 2543 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 29 00:54:10.053389 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.052519 2543 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-136.ec2.internal.18c69c914525f7a8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-136.ec2.internal,UID:ip-10-0-132-136.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-136.ec2.internal,},FirstTimestamp:2026-07-29 00:54:10.043287464 +0000 UTC m=+0.313931761,LastTimestamp:2026-07-29 00:54:10.043287464 +0000 UTC m=+0.313931761,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-136.ec2.internal,}" Jul 29 00:54:10.054018 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.054003 2543 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 29 00:54:10.054118 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.054054 2543 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 29 00:54:10.054118 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.054085 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-136.ec2.internal\" not found" Jul 29 00:54:10.054783 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.054763 2543 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 29 00:54:10.054875 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.054786 2543 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 29 00:54:10.054875 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.054764 2543 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 29 00:54:10.055244 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.055225 2543 reconstruct.go:97] "Volume reconstruction finished" Jul 29 00:54:10.055244 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.055238 2543 factory.go:153] Registering CRI-O factory Jul 29 00:54:10.055350 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.055253 2543 factory.go:223] Registration of the crio container factory successfully Jul 29 00:54:10.055350 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.055292 2543 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 29 00:54:10.055350 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.055299 2543 factory.go:55] Registering systemd factory Jul 29 00:54:10.055350 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.055307 2543 factory.go:223] Registration of the systemd container factory successfully Jul 29 00:54:10.055350 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.055241 2543 reconciler.go:29] "Reconciler: start to sync state" Jul 29 00:54:10.055350 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.055326 2543 factory.go:103] Registering Raw factory Jul 29 00:54:10.055350 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.055336 2543 manager.go:1196] Started watching for new ooms in manager Jul 29 00:54:10.055664 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.055648 2543 manager.go:349] Starting recovery of all containers Jul 29 00:54:10.065797 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.065766 2543 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 29 00:54:10.065929 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.065903 2543 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-136.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 29 00:54:10.068204 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.068188 2543 manager.go:354] Recovery completed Jul 29 00:54:10.073343 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.073329 2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 00:54:10.076313 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.076295 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-136.ec2.internal" event="NodeHasSufficientMemory" Jul 29 00:54:10.076392 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.076323 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-136.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 00:54:10.076392 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.076333 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-136.ec2.internal" event="NodeHasSufficientPID" Jul 29 00:54:10.076825 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.076794 2543 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 29 00:54:10.076825 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.076810 2543 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 29 00:54:10.076949 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.076846 2543 state_mem.go:36] "Initialized new in-memory state store" Jul 29 00:54:10.076949 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.076851 2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 00:54:10.077790 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.077776 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-136.ec2.internal" event="NodeHasSufficientMemory" Jul 29 00:54:10.077888 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.077807 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-136.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 00:54:10.077888 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.077836 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-136.ec2.internal" event="NodeHasSufficientPID" Jul 29 00:54:10.080546 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.080530 2543 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5gmjg" Jul 29 00:54:10.086181 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.086092 2543 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-136.ec2.internal.18c69c91471ddb53 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-136.ec2.internal,UID:ip-10-0-132-136.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-136.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-136.ec2.internal,},FirstTimestamp:2026-07-29 00:54:10.076310355 +0000 UTC m=+0.346954653,LastTimestamp:2026-07-29 00:54:10.076310355 +0000 UTC m=+0.346954653,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-136.ec2.internal,}" Jul 29 00:54:10.087018 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.087001 2543 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5gmjg" Jul 29 00:54:10.087516 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.087448 2543 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-136.ec2.internal.18c69c91471e1daa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-136.ec2.internal,UID:ip-10-0-132-136.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-136.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-136.ec2.internal,},FirstTimestamp:2026-07-29 00:54:10.076327338 +0000 UTC m=+0.346971635,LastTimestamp:2026-07-29 00:54:10.076327338 +0000 UTC m=+0.346971635,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-136.ec2.internal,}" Jul 29 00:54:10.096156 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.096085 2543 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-136.ec2.internal.18c69c91471e42fb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-136.ec2.internal,UID:ip-10-0-132-136.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-136.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-136.ec2.internal,},FirstTimestamp:2026-07-29 00:54:10.076336891 +0000 UTC m=+0.346981190,LastTimestamp:2026-07-29 00:54:10.076336891 +0000 UTC m=+0.346981190,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-136.ec2.internal,}" Jul 29 00:54:10.106998 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.106922 2543 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-136.ec2.internal.18c69c91471ddb53\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-136.ec2.internal.18c69c91471ddb53 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-136.ec2.internal,UID:ip-10-0-132-136.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-136.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-136.ec2.internal,},FirstTimestamp:2026-07-29 00:54:10.076310355 +0000 UTC m=+0.346954653,LastTimestamp:2026-07-29 00:54:10.077789918 +0000 UTC m=+0.348434221,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-136.ec2.internal,}" Jul 29 00:54:10.115714 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.115654 2543 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-136.ec2.internal.18c69c91471e1daa\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-136.ec2.internal.18c69c91471e1daa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-136.ec2.internal,UID:ip-10-0-132-136.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-136.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-136.ec2.internal,},FirstTimestamp:2026-07-29 00:54:10.076327338 +0000 UTC m=+0.346971635,LastTimestamp:2026-07-29 00:54:10.077826824 +0000 UTC m=+0.348471123,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-136.ec2.internal,}" Jul 29 00:54:10.130448 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.130386 2543 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-136.ec2.internal.18c69c91471e42fb\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-136.ec2.internal.18c69c91471e42fb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-136.ec2.internal,UID:ip-10-0-132-136.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-136.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-136.ec2.internal,},FirstTimestamp:2026-07-29 00:54:10.076336891 +0000 UTC m=+0.346981190,LastTimestamp:2026-07-29 00:54:10.077842368 +0000 UTC m=+0.348486669,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-136.ec2.internal,}" Jul 29 00:54:10.154515 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.154496 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-136.ec2.internal\" not found" Jul 29 00:54:10.224918 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.224896 2543 policy_none.go:49] "None policy: Start" Jul 29 00:54:10.224918 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.224920 2543 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 29 00:54:10.225052 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.224934 2543 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 29 00:54:10.225920 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.225909 2543 policy_none.go:47] "Start" Jul 29 00:54:10.255282 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.255258 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-136.ec2.internal\" not found" Jul 29 00:54:10.255356 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.255346 2543 manager.go:337] "Starting Device Plugin manager" Jul 29 00:54:10.255388 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.255373 2543 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 29 00:54:10.255388 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.255383 2543 server.go:87] "Starting device plugin registration server" Jul 29 00:54:10.255638 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.255626 2543 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 29 00:54:10.255683 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.255642 2543 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 29 00:54:10.255761 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.255745 2543 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 29 00:54:10.255852 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.255840 2543 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 29 00:54:10.255913 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.255855 2543 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 29 00:54:10.256944 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.256756 2543 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 29 00:54:10.257009 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.256975 2543 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-136.ec2.internal\" not found" Jul 29 00:54:10.272968 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.272950 2543 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-136.ec2.internal\" not found" node="ip-10-0-132-136.ec2.internal" Jul 29 00:54:10.335537 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.335505 2543 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 29 00:54:10.336758 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.336735 2543 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 29 00:54:10.336870 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.336776 2543 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 29 00:54:10.336870 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.336805 2543 kubelet.go:2448] "Starting kubelet main sync loop" Jul 29 00:54:10.336976 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.336863 2543 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 29 00:54:10.339794 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.339777 2543 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 00:54:10.355893 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.355862 2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 00:54:10.356728 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.356712 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-136.ec2.internal" event="NodeHasSufficientMemory" Jul 29 00:54:10.356796 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.356747 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-136.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 00:54:10.356796 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.356762 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-136.ec2.internal" event="NodeHasSufficientPID" Jul 29 00:54:10.356796 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.356790 2543 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-136.ec2.internal" Jul 29 00:54:10.362973 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.362959 2543 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-136.ec2.internal" Jul 29 00:54:10.363013 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.362979 2543 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-136.ec2.internal\": node \"ip-10-0-132-136.ec2.internal\" not found" Jul 29 00:54:10.386923 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.386905 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-136.ec2.internal\" not found" Jul 29 00:54:10.437531 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.437507 2543 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-136.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-136.ec2.internal"] Jul 29 00:54:10.437595 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.437570 2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 00:54:10.438447 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.438435 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-136.ec2.internal" event="NodeHasSufficientMemory" Jul 29 00:54:10.438505 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.438463 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-136.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 00:54:10.438505 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.438475 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-136.ec2.internal" event="NodeHasSufficientPID" Jul 29 00:54:10.439703 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.439692 2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 00:54:10.439808 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.439796 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-136.ec2.internal" Jul 29 00:54:10.439862 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.439841 2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 00:54:10.440375 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.440359 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-136.ec2.internal" event="NodeHasSufficientMemory" Jul 29 00:54:10.440419 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.440374 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-136.ec2.internal" event="NodeHasSufficientMemory" Jul 29 00:54:10.440419 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.440387 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-136.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 00:54:10.440419 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.440394 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-136.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 00:54:10.440419 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.440401 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-136.ec2.internal" event="NodeHasSufficientPID" Jul 29 00:54:10.440419 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.440403 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-136.ec2.internal" event="NodeHasSufficientPID" Jul 29 00:54:10.441494 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.441482 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-136.ec2.internal" Jul 29 00:54:10.441540 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.441511 2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 00:54:10.442249 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.442235 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-136.ec2.internal" event="NodeHasSufficientMemory" Jul 29 00:54:10.442333 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.442264 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-136.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 00:54:10.442333 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.442279 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-136.ec2.internal" event="NodeHasSufficientPID" Jul 29 00:54:10.455987 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.455969 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e1b4121d255f123d5bc2e0fe24413a8b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-136.ec2.internal\" (UID: \"e1b4121d255f123d5bc2e0fe24413a8b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-136.ec2.internal" Jul 29 00:54:10.456073 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.455995 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c032dfa2331a0d7e82f450dc05e5aa6c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-136.ec2.internal\" (UID: \"c032dfa2331a0d7e82f450dc05e5aa6c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-136.ec2.internal" Jul 29 00:54:10.456073 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.456014 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c032dfa2331a0d7e82f450dc05e5aa6c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-136.ec2.internal\" (UID: \"c032dfa2331a0d7e82f450dc05e5aa6c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-136.ec2.internal" Jul 29 00:54:10.473511 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.473490 2543 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-136.ec2.internal\" not found" node="ip-10-0-132-136.ec2.internal" Jul 29 00:54:10.477880 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.477864 2543 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-136.ec2.internal\" not found" node="ip-10-0-132-136.ec2.internal" Jul 29 00:54:10.487884 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.487841 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-136.ec2.internal\" not found" Jul 29 00:54:10.556609 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.556584 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e1b4121d255f123d5bc2e0fe24413a8b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-136.ec2.internal\" (UID: \"e1b4121d255f123d5bc2e0fe24413a8b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-136.ec2.internal" Jul 29 00:54:10.556609 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.556609 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c032dfa2331a0d7e82f450dc05e5aa6c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-136.ec2.internal\" (UID: \"c032dfa2331a0d7e82f450dc05e5aa6c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-136.ec2.internal" Jul 29 00:54:10.556754 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.556634 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c032dfa2331a0d7e82f450dc05e5aa6c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-136.ec2.internal\" (UID: \"c032dfa2331a0d7e82f450dc05e5aa6c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-136.ec2.internal" Jul 29 00:54:10.556754 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.556688 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c032dfa2331a0d7e82f450dc05e5aa6c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-136.ec2.internal\" (UID: \"c032dfa2331a0d7e82f450dc05e5aa6c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-136.ec2.internal" Jul 29 00:54:10.556754 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.556691 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e1b4121d255f123d5bc2e0fe24413a8b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-136.ec2.internal\" (UID: \"e1b4121d255f123d5bc2e0fe24413a8b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-136.ec2.internal" Jul 29 00:54:10.556754 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.556704 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c032dfa2331a0d7e82f450dc05e5aa6c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-136.ec2.internal\" (UID: \"c032dfa2331a0d7e82f450dc05e5aa6c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-136.ec2.internal" Jul 29 00:54:10.588706 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.588685 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-136.ec2.internal\" not found" Jul 29 00:54:10.689191 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.689162 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-136.ec2.internal\" not found" Jul 29 00:54:10.775769 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.775704 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-136.ec2.internal" Jul 29 00:54:10.781482 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.781460 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-136.ec2.internal" Jul 29 00:54:10.789594 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.789576 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-136.ec2.internal\" not found" Jul 29 00:54:10.890145 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:10.890108 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-136.ec2.internal\" not found" Jul 29 00:54:10.921694 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.921672 2543 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 00:54:10.954826 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.954793 2543 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-136.ec2.internal" Jul 29 00:54:10.967428 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.967411 2543 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 00:54:10.969252 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.969241 2543 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-136.ec2.internal" Jul 29 00:54:10.977380 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:10.977365 2543 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 00:54:11.000461 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.000443 2543 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 29 00:54:11.000559 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.000542 2543 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 00:54:11.000602 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.000581 2543 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 00:54:11.033748 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.033684 2543 apiserver.go:52] "Watching apiserver" Jul 29 00:54:11.049762 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.049739 2543 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 29 00:54:11.050149 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.050125 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-h8nv4","kube-system/global-pull-secret-syncer-lf9gq","kube-system/konnectivity-agent-j7xsc","openshift-cluster-node-tuning-operator/tuned-jh44t","openshift-dns/node-resolver-thq5l","openshift-image-registry/node-ca-nqv9s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-136.ec2.internal","openshift-multus/network-metrics-daemon-jx7tt","openshift-network-operator/iptables-alerter-lsz6j","kube-system/kube-apiserver-proxy-ip-10-0-132-136.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z","openshift-multus/multus-2mrv7","openshift-multus/multus-additional-cni-plugins-d7g65","openshift-network-diagnostics/network-check-target-g87tb"] Jul 29 00:54:11.051719 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.051703 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.052822 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.052800 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:11.052923 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:11.052891 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g87tb" podUID="926d37fa-f392-43d9-b446-160238607eff" Jul 29 00:54:11.053849 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.053828 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 29 00:54:11.053985 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.053966 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j7xsc" Jul 29 00:54:11.054152 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.054121 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 29 00:54:11.054222 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.054153 2543 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 29 00:54:11.054222 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.054169 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 29 00:54:11.054316 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.054272 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vc797\"" Jul 29 00:54:11.055223 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.055203 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.056045 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.056022 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 29 00:54:11.056141 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.056120 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 29 00:54:11.056241 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.056218 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-98hxg\"" Jul 29 00:54:11.056344 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.056247 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-thq5l" Jul 29 00:54:11.057164 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.057150 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 29 00:54:11.057223 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.057152 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 29 00:54:11.057223 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.057204 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9sk8x\"" Jul 29 00:54:11.057675 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.057655 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nqv9s" Jul 29 00:54:11.058403 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.058374 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 29 00:54:11.058504 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.058445 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 29 00:54:11.058504 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.058467 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sgs2g\"" Jul 29 00:54:11.059118 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059100 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.059208 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059102 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e79868cd-1ac6-4262-ab1f-687d466f8b4d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-68v7z\" (UID: \"e79868cd-1ac6-4262-ab1f-687d466f8b4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.059208 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059205 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-etc-sysconfig\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.059324 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059224 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-sys\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.059324 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059248 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-etc-tuned\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.059324 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059291 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-tmp\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.059478 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059339 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e79868cd-1ac6-4262-ab1f-687d466f8b4d-etc-selinux\") pod \"aws-ebs-csi-driver-node-68v7z\" (UID: \"e79868cd-1ac6-4262-ab1f-687d466f8b4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.059478 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059366 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b304c44-8f9d-4830-a2e7-d5559eeb879f-agent-certs\") pod \"konnectivity-agent-j7xsc\" (UID: \"0b304c44-8f9d-4830-a2e7-d5559eeb879f\") " pod="kube-system/konnectivity-agent-j7xsc" Jul 29 00:54:11.059478 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059397 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b304c44-8f9d-4830-a2e7-d5559eeb879f-konnectivity-ca\") pod \"konnectivity-agent-j7xsc\" (UID: \"0b304c44-8f9d-4830-a2e7-d5559eeb879f\") " pod="kube-system/konnectivity-agent-j7xsc" Jul 29 00:54:11.059478 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059425 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-etc-kubernetes\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.059478 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059455 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-etc-sysctl-d\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.059478 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059481 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-etc-systemd\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.059775 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059497 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-run\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.059775 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059521 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml2mc\" (UniqueName: \"kubernetes.io/projected/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-kube-api-access-ml2mc\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.059775 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059545 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhbwm\" (UniqueName: \"kubernetes.io/projected/5719a514-b8de-4661-ab2a-fe0587726ae8-kube-api-access-rhbwm\") pod \"node-resolver-thq5l\" (UID: \"5719a514-b8de-4661-ab2a-fe0587726ae8\") " pod="openshift-dns/node-resolver-thq5l" Jul 29 00:54:11.059775 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059599 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-etc-modprobe-d\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.059775 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059621 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5719a514-b8de-4661-ab2a-fe0587726ae8-hosts-file\") pod \"node-resolver-thq5l\" (UID: \"5719a514-b8de-4661-ab2a-fe0587726ae8\") " pod="openshift-dns/node-resolver-thq5l" Jul 29 00:54:11.059775 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059659 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e79868cd-1ac6-4262-ab1f-687d466f8b4d-socket-dir\") pod \"aws-ebs-csi-driver-node-68v7z\" (UID: \"e79868cd-1ac6-4262-ab1f-687d466f8b4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.059775 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059695 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-host\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.059775 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059753 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5719a514-b8de-4661-ab2a-fe0587726ae8-tmp-dir\") pod \"node-resolver-thq5l\" (UID: \"5719a514-b8de-4661-ab2a-fe0587726ae8\") " pod="openshift-dns/node-resolver-thq5l" Jul 29 00:54:11.060131 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059781 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e79868cd-1ac6-4262-ab1f-687d466f8b4d-sys-fs\") pod \"aws-ebs-csi-driver-node-68v7z\" (UID: \"e79868cd-1ac6-4262-ab1f-687d466f8b4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.060131 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059805 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jspd\" (UniqueName: \"kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd\") pod \"network-check-target-g87tb\" (UID: \"926d37fa-f392-43d9-b446-160238607eff\") " pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:11.060131 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059846 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-lib-modules\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.060131 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059864 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 29 00:54:11.060131 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059866 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e79868cd-1ac6-4262-ab1f-687d466f8b4d-registration-dir\") pod \"aws-ebs-csi-driver-node-68v7z\" (UID: \"e79868cd-1ac6-4262-ab1f-687d466f8b4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.060131 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059891 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mdv9\" (UniqueName: \"kubernetes.io/projected/e79868cd-1ac6-4262-ab1f-687d466f8b4d-kube-api-access-6mdv9\") pod \"aws-ebs-csi-driver-node-68v7z\" (UID: \"e79868cd-1ac6-4262-ab1f-687d466f8b4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.060131 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059914 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-etc-sysctl-conf\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.060131 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059931 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e79868cd-1ac6-4262-ab1f-687d466f8b4d-device-dir\") pod \"aws-ebs-csi-driver-node-68v7z\" (UID: \"e79868cd-1ac6-4262-ab1f-687d466f8b4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.060131 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.059946 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-var-lib-kubelet\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.060131 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.060132 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 29 00:54:11.060513 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.060155 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 29 00:54:11.060513 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.060184 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6d2n4\"" Jul 29 00:54:11.060513 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.060410 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.061353 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.061333 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 29 00:54:11.061443 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.061338 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hrdgv\"" Jul 29 00:54:11.061443 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.061433 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 29 00:54:11.061549 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.061488 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 29 00:54:11.061698 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.061682 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.062086 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.062067 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 29 00:54:11.062205 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.062182 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 29 00:54:11.062281 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.062232 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 29 00:54:11.062347 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.062333 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 29 00:54:11.062347 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.062339 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 29 00:54:11.062449 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.062363 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pthxm\"" Jul 29 00:54:11.062729 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.062702 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 29 00:54:11.062868 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.062848 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 29 00:54:11.062974 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.062849 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 29 00:54:11.063159 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.063139 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lsz6j" Jul 29 00:54:11.064087 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.064071 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 29 00:54:11.064087 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.064085 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7t24q\"" Jul 29 00:54:11.064717 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.064699 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:11.064795 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:11.064770 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jx7tt" podUID="78bcb31a-2890-4527-b8f0-73cabeaecd3e" Jul 29 00:54:11.065079 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.065062 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 29 00:54:11.065135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.065072 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 29 00:54:11.065192 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.065165 2543 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 00:54:11.065364 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.065349 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k9gmb\"" Jul 29 00:54:11.065436 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.065419 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 29 00:54:11.066501 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.066375 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:11.066501 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:11.066446 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lf9gq" podUID="03f35a91-81af-4c82-976b-50ad3b9d4ea6" Jul 29 00:54:11.087521 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.087499 2543 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4k9ht" Jul 29 00:54:11.092884 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.092845 2543 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4k9ht" Jul 29 00:54:11.156174 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.156150 2543 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 29 00:54:11.161020 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161000 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-host-run-ovn-kubernetes\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.161102 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161030 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-host-cni-netd\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.161102 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161058 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mdv9\" (UniqueName: \"kubernetes.io/projected/e79868cd-1ac6-4262-ab1f-687d466f8b4d-kube-api-access-6mdv9\") pod \"aws-ebs-csi-driver-node-68v7z\" (UID: \"e79868cd-1ac6-4262-ab1f-687d466f8b4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.161102 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161079 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f974fab-a19b-419d-bc9f-ff551c5d928f-cnibin\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.161236 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161116 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f974fab-a19b-419d-bc9f-ff551c5d928f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.161236 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161149 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e79868cd-1ac6-4262-ab1f-687d466f8b4d-device-dir\") pod \"aws-ebs-csi-driver-node-68v7z\" (UID: \"e79868cd-1ac6-4262-ab1f-687d466f8b4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.161236 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161176 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-multus-socket-dir-parent\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.161236 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161194 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-host-run-netns\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.161236 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161211 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1f974fab-a19b-419d-bc9f-ff551c5d928f-os-release\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.161236 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161231 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/47a76b9a-5cc4-4739-8ab3-2745dab6d404-iptables-alerter-script\") pod \"iptables-alerter-lsz6j\" (UID: \"47a76b9a-5cc4-4739-8ab3-2745dab6d404\") " pod="openshift-network-operator/iptables-alerter-lsz6j" Jul 29 00:54:11.161445 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161234 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e79868cd-1ac6-4262-ab1f-687d466f8b4d-device-dir\") pod \"aws-ebs-csi-driver-node-68v7z\" (UID: \"e79868cd-1ac6-4262-ab1f-687d466f8b4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.161445 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161267 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-var-lib-kubelet\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.161445 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161299 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwzct\" (UniqueName: \"kubernetes.io/projected/a2127c7c-06fc-4849-ab98-2975c0862540-kube-api-access-mwzct\") pod \"node-ca-nqv9s\" (UID: \"a2127c7c-06fc-4849-ab98-2975c0862540\") " pod="openshift-image-registry/node-ca-nqv9s" Jul 29 00:54:11.161445 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161326 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-system-cni-dir\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.161445 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161348 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-var-lib-kubelet\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.161445 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161352 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07544af8-8a7f-4ada-add4-28bc48719e32-multus-daemon-config\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.161445 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161383 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a557028-eba5-431a-8690-dd95f88e82df-env-overrides\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.161445 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161411 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a557028-eba5-431a-8690-dd95f88e82df-ovnkube-script-lib\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.161445 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161438 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4ctw\" (UniqueName: \"kubernetes.io/projected/1f974fab-a19b-419d-bc9f-ff551c5d928f-kube-api-access-l4ctw\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.161863 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161468 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-sys\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.161863 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161494 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-etc-tuned\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.161863 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161546 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-sys\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.161863 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161559 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-tmp\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.161863 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161581 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e79868cd-1ac6-4262-ab1f-687d466f8b4d-etc-selinux\") pod \"aws-ebs-csi-driver-node-68v7z\" (UID: \"e79868cd-1ac6-4262-ab1f-687d466f8b4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.161863 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161636 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2127c7c-06fc-4849-ab98-2975c0862540-serviceca\") pod \"node-ca-nqv9s\" (UID: \"a2127c7c-06fc-4849-ab98-2975c0862540\") " pod="openshift-image-registry/node-ca-nqv9s" Jul 29 00:54:11.161863 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161701 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-host-var-lib-cni-multus\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.161863 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161720 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e79868cd-1ac6-4262-ab1f-687d466f8b4d-etc-selinux\") pod \"aws-ebs-csi-driver-node-68v7z\" (UID: \"e79868cd-1ac6-4262-ab1f-687d466f8b4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.161863 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161739 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f974fab-a19b-419d-bc9f-ff551c5d928f-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.161863 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161777 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b304c44-8f9d-4830-a2e7-d5559eeb879f-agent-certs\") pod \"konnectivity-agent-j7xsc\" (UID: \"0b304c44-8f9d-4830-a2e7-d5559eeb879f\") " pod="kube-system/konnectivity-agent-j7xsc" Jul 29 00:54:11.161863 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161822 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-etc-sysctl-d\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.162396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161858 2543 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 29 00:54:11.162396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161881 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-run\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.162396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161909 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml2mc\" (UniqueName: \"kubernetes.io/projected/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-kube-api-access-ml2mc\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.162396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161912 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-etc-sysctl-d\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.162396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161961 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-multus-conf-dir\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.162396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.161990 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-run\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.162396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162083 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-host-run-multus-certs\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.162396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162112 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-run-ovn\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.162396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162138 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmmk7\" (UniqueName: \"kubernetes.io/projected/8a557028-eba5-431a-8690-dd95f88e82df-kube-api-access-wmmk7\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.162396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162187 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-etc-modprobe-d\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.162396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162223 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e79868cd-1ac6-4262-ab1f-687d466f8b4d-socket-dir\") pod \"aws-ebs-csi-driver-node-68v7z\" (UID: \"e79868cd-1ac6-4262-ab1f-687d466f8b4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.162396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162253 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-systemd-units\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.162396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162282 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5719a514-b8de-4661-ab2a-fe0587726ae8-tmp-dir\") pod \"node-resolver-thq5l\" (UID: \"5719a514-b8de-4661-ab2a-fe0587726ae8\") " pod="openshift-dns/node-resolver-thq5l" Jul 29 00:54:11.162396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162308 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e79868cd-1ac6-4262-ab1f-687d466f8b4d-sys-fs\") pod \"aws-ebs-csi-driver-node-68v7z\" (UID: \"e79868cd-1ac6-4262-ab1f-687d466f8b4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.162396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162317 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-etc-modprobe-d\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.162396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162332 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-cnibin\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.162396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162351 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e79868cd-1ac6-4262-ab1f-687d466f8b4d-socket-dir\") pod \"aws-ebs-csi-driver-node-68v7z\" (UID: \"e79868cd-1ac6-4262-ab1f-687d466f8b4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.162396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162356 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret\") pod \"global-pull-secret-syncer-lf9gq\" (UID: \"03f35a91-81af-4c82-976b-50ad3b9d4ea6\") " pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:11.163244 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162384 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-host-kubelet\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.163244 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162409 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f974fab-a19b-419d-bc9f-ff551c5d928f-system-cni-dir\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.163244 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162435 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e79868cd-1ac6-4262-ab1f-687d466f8b4d-sys-fs\") pod \"aws-ebs-csi-driver-node-68v7z\" (UID: \"e79868cd-1ac6-4262-ab1f-687d466f8b4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.163244 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162435 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-host-run-k8s-cni-cncf-io\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.163244 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162483 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03f35a91-81af-4c82-976b-50ad3b9d4ea6-kubelet-config\") pod \"global-pull-secret-syncer-lf9gq\" (UID: \"03f35a91-81af-4c82-976b-50ad3b9d4ea6\") " pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:11.163244 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162525 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-host-cni-bin\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.163244 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162563 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jspd\" (UniqueName: \"kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd\") pod \"network-check-target-g87tb\" (UID: \"926d37fa-f392-43d9-b446-160238607eff\") " pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:11.163244 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162590 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5719a514-b8de-4661-ab2a-fe0587726ae8-tmp-dir\") pod \"node-resolver-thq5l\" (UID: \"5719a514-b8de-4661-ab2a-fe0587726ae8\") " pod="openshift-dns/node-resolver-thq5l" Jul 29 00:54:11.163244 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162596 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-lib-modules\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.163244 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162707 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e79868cd-1ac6-4262-ab1f-687d466f8b4d-registration-dir\") pod \"aws-ebs-csi-driver-node-68v7z\" (UID: \"e79868cd-1ac6-4262-ab1f-687d466f8b4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.163244 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162763 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a557028-eba5-431a-8690-dd95f88e82df-ovn-node-metrics-cert\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.163244 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162782 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-lib-modules\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.163244 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162788 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-etc-sysctl-conf\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.163244 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162804 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e79868cd-1ac6-4262-ab1f-687d466f8b4d-registration-dir\") pod \"aws-ebs-csi-driver-node-68v7z\" (UID: \"e79868cd-1ac6-4262-ab1f-687d466f8b4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.163244 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162852 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b5qg\" (UniqueName: \"kubernetes.io/projected/78bcb31a-2890-4527-b8f0-73cabeaecd3e-kube-api-access-7b5qg\") pod \"network-metrics-daemon-jx7tt\" (UID: \"78bcb31a-2890-4527-b8f0-73cabeaecd3e\") " pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:11.163244 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162880 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2127c7c-06fc-4849-ab98-2975c0862540-host\") pod \"node-ca-nqv9s\" (UID: \"a2127c7c-06fc-4849-ab98-2975c0862540\") " pod="openshift-image-registry/node-ca-nqv9s" Jul 29 00:54:11.163244 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162883 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-etc-sysctl-conf\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.164140 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162919 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-host-run-netns\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.164140 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.162980 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-log-socket\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.164140 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163009 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e79868cd-1ac6-4262-ab1f-687d466f8b4d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-68v7z\" (UID: \"e79868cd-1ac6-4262-ab1f-687d466f8b4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.164140 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163034 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs\") pod \"network-metrics-daemon-jx7tt\" (UID: \"78bcb31a-2890-4527-b8f0-73cabeaecd3e\") " pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:11.164140 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163060 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-multus-cni-dir\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.164140 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163083 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-os-release\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.164140 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163109 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-run-openvswitch\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.164140 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163112 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e79868cd-1ac6-4262-ab1f-687d466f8b4d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-68v7z\" (UID: \"e79868cd-1ac6-4262-ab1f-687d466f8b4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.164140 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163136 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-node-log\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.164140 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163164 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f974fab-a19b-419d-bc9f-ff551c5d928f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.164140 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163188 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-host-var-lib-kubelet\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.164140 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163213 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-etc-sysconfig\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.164140 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163246 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-run-systemd\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.164140 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163273 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-var-lib-openvswitch\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.164140 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163300 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.164140 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163329 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b304c44-8f9d-4830-a2e7-d5559eeb879f-konnectivity-ca\") pod \"konnectivity-agent-j7xsc\" (UID: \"0b304c44-8f9d-4830-a2e7-d5559eeb879f\") " pod="kube-system/konnectivity-agent-j7xsc" Jul 29 00:54:11.164140 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163332 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-etc-sysconfig\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.165009 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163364 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-etc-kubernetes\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.165009 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163394 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-etc-systemd\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.165009 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163443 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhbwm\" (UniqueName: \"kubernetes.io/projected/5719a514-b8de-4661-ab2a-fe0587726ae8-kube-api-access-rhbwm\") pod \"node-resolver-thq5l\" (UID: \"5719a514-b8de-4661-ab2a-fe0587726ae8\") " pod="openshift-dns/node-resolver-thq5l" Jul 29 00:54:11.165009 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163472 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-host-var-lib-cni-bin\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.165009 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163497 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-etc-kubernetes\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.165009 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163523 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5719a514-b8de-4661-ab2a-fe0587726ae8-hosts-file\") pod \"node-resolver-thq5l\" (UID: \"5719a514-b8de-4661-ab2a-fe0587726ae8\") " pod="openshift-dns/node-resolver-thq5l" Jul 29 00:54:11.165009 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163546 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-hostroot\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.165009 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163573 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a557028-eba5-431a-8690-dd95f88e82df-ovnkube-config\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.165009 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163612 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f974fab-a19b-419d-bc9f-ff551c5d928f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.165009 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163645 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47a76b9a-5cc4-4739-8ab3-2745dab6d404-host-slash\") pod \"iptables-alerter-lsz6j\" (UID: \"47a76b9a-5cc4-4739-8ab3-2745dab6d404\") " pod="openshift-network-operator/iptables-alerter-lsz6j" Jul 29 00:54:11.165009 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163676 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-host\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.165009 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163702 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07544af8-8a7f-4ada-add4-28bc48719e32-cni-binary-copy\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.165009 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163736 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03f35a91-81af-4c82-976b-50ad3b9d4ea6-dbus\") pod \"global-pull-secret-syncer-lf9gq\" (UID: \"03f35a91-81af-4c82-976b-50ad3b9d4ea6\") " pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:11.165009 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163760 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-host-slash\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.165009 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163783 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-etc-openvswitch\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.165009 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163882 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh7nc\" (UniqueName: \"kubernetes.io/projected/47a76b9a-5cc4-4739-8ab3-2745dab6d404-kube-api-access-vh7nc\") pod \"iptables-alerter-lsz6j\" (UID: \"47a76b9a-5cc4-4739-8ab3-2745dab6d404\") " pod="openshift-network-operator/iptables-alerter-lsz6j" Jul 29 00:54:11.165009 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163886 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-etc-kubernetes\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.165596 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163915 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd7x9\" (UniqueName: \"kubernetes.io/projected/07544af8-8a7f-4ada-add4-28bc48719e32-kube-api-access-dd7x9\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.165596 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163951 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-host\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.165596 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.163999 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-etc-systemd\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.165596 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.164036 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5719a514-b8de-4661-ab2a-fe0587726ae8-hosts-file\") pod \"node-resolver-thq5l\" (UID: \"5719a514-b8de-4661-ab2a-fe0587726ae8\") " pod="openshift-dns/node-resolver-thq5l" Jul 29 00:54:11.165596 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.164976 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b304c44-8f9d-4830-a2e7-d5559eeb879f-konnectivity-ca\") pod \"konnectivity-agent-j7xsc\" (UID: \"0b304c44-8f9d-4830-a2e7-d5559eeb879f\") " pod="kube-system/konnectivity-agent-j7xsc" Jul 29 00:54:11.165596 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.165250 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-etc-tuned\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.165596 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.165351 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-tmp\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.165596 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.165560 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b304c44-8f9d-4830-a2e7-d5559eeb879f-agent-certs\") pod \"konnectivity-agent-j7xsc\" (UID: \"0b304c44-8f9d-4830-a2e7-d5559eeb879f\") " pod="kube-system/konnectivity-agent-j7xsc" Jul 29 00:54:11.168931 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.168911 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mdv9\" (UniqueName: \"kubernetes.io/projected/e79868cd-1ac6-4262-ab1f-687d466f8b4d-kube-api-access-6mdv9\") pod \"aws-ebs-csi-driver-node-68v7z\" (UID: \"e79868cd-1ac6-4262-ab1f-687d466f8b4d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.172090 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:11.172069 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 00:54:11.172225 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:11.172094 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 00:54:11.172225 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:11.172108 2543 projected.go:196] Error preparing data for projected volume kube-api-access-5jspd for pod openshift-network-diagnostics/network-check-target-g87tb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 00:54:11.172225 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:11.172180 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd podName:926d37fa-f392-43d9-b446-160238607eff nodeName:}" failed. No retries permitted until 2026-07-29 00:54:11.672158286 +0000 UTC m=+1.942802591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5jspd" (UniqueName: "kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd") pod "network-check-target-g87tb" (UID: "926d37fa-f392-43d9-b446-160238607eff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 00:54:11.173139 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.173120 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml2mc\" (UniqueName: \"kubernetes.io/projected/11a3edfc-262f-4bb8-96fb-4b3b2542d9af-kube-api-access-ml2mc\") pod \"tuned-jh44t\" (UID: \"11a3edfc-262f-4bb8-96fb-4b3b2542d9af\") " pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.174457 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.174433 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhbwm\" (UniqueName: \"kubernetes.io/projected/5719a514-b8de-4661-ab2a-fe0587726ae8-kube-api-access-rhbwm\") pod \"node-resolver-thq5l\" (UID: \"5719a514-b8de-4661-ab2a-fe0587726ae8\") " pod="openshift-dns/node-resolver-thq5l" Jul 29 00:54:11.193314 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.193290 2543 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 00:54:11.227715 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.227659 2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 00:49:10 +0000 UTC" deadline="2028-05-06 19:37:12.23196986 +0000 UTC" Jul 29 00:54:11.227715 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.227707 2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15546h43m1.004265486s" Jul 29 00:54:11.264693 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.264658 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/47a76b9a-5cc4-4739-8ab3-2745dab6d404-iptables-alerter-script\") pod \"iptables-alerter-lsz6j\" (UID: \"47a76b9a-5cc4-4739-8ab3-2745dab6d404\") " pod="openshift-network-operator/iptables-alerter-lsz6j" Jul 29 00:54:11.264693 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.264698 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwzct\" (UniqueName: \"kubernetes.io/projected/a2127c7c-06fc-4849-ab98-2975c0862540-kube-api-access-mwzct\") pod \"node-ca-nqv9s\" (UID: \"a2127c7c-06fc-4849-ab98-2975c0862540\") " pod="openshift-image-registry/node-ca-nqv9s" Jul 29 00:54:11.264976 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.264716 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-system-cni-dir\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.264976 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.264734 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07544af8-8a7f-4ada-add4-28bc48719e32-multus-daemon-config\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.264976 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.264758 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a557028-eba5-431a-8690-dd95f88e82df-env-overrides\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.264976 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.264857 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a557028-eba5-431a-8690-dd95f88e82df-ovnkube-script-lib\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.264976 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.264905 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4ctw\" (UniqueName: \"kubernetes.io/projected/1f974fab-a19b-419d-bc9f-ff551c5d928f-kube-api-access-l4ctw\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.264976 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.264953 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2127c7c-06fc-4849-ab98-2975c0862540-serviceca\") pod \"node-ca-nqv9s\" (UID: \"a2127c7c-06fc-4849-ab98-2975c0862540\") " pod="openshift-image-registry/node-ca-nqv9s" Jul 29 00:54:11.264976 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.264865 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-system-cni-dir\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.265349 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.264983 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-host-var-lib-cni-multus\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.265349 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265009 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f974fab-a19b-419d-bc9f-ff551c5d928f-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.265349 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265069 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-host-var-lib-cni-multus\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.265523 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265493 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a557028-eba5-431a-8690-dd95f88e82df-env-overrides\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.265584 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265537 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/47a76b9a-5cc4-4739-8ab3-2745dab6d404-iptables-alerter-script\") pod \"iptables-alerter-lsz6j\" (UID: \"47a76b9a-5cc4-4739-8ab3-2745dab6d404\") " pod="openshift-network-operator/iptables-alerter-lsz6j" Jul 29 00:54:11.265584 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265560 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-multus-conf-dir\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.265584 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265562 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07544af8-8a7f-4ada-add4-28bc48719e32-multus-daemon-config\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.265584 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265581 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-host-run-multus-certs\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.265802 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265599 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-run-ovn\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.265802 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265617 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmmk7\" (UniqueName: \"kubernetes.io/projected/8a557028-eba5-431a-8690-dd95f88e82df-kube-api-access-wmmk7\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.265802 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265627 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-multus-conf-dir\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.265802 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265638 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f974fab-a19b-419d-bc9f-ff551c5d928f-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.265802 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265651 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-host-run-multus-certs\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.265802 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265643 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-systemd-units\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.265802 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265675 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-run-ovn\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.265802 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265700 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-cnibin\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.265802 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265717 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-systemd-units\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.265802 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265727 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret\") pod \"global-pull-secret-syncer-lf9gq\" (UID: \"03f35a91-81af-4c82-976b-50ad3b9d4ea6\") " pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:11.265802 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265753 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-host-kubelet\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.265802 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265777 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f974fab-a19b-419d-bc9f-ff551c5d928f-system-cni-dir\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.265802 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265769 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-cnibin\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.265802 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265802 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-host-run-k8s-cni-cncf-io\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.266559 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265837 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03f35a91-81af-4c82-976b-50ad3b9d4ea6-kubelet-config\") pod \"global-pull-secret-syncer-lf9gq\" (UID: \"03f35a91-81af-4c82-976b-50ad3b9d4ea6\") " pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:11.266559 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265847 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a2127c7c-06fc-4849-ab98-2975c0862540-serviceca\") pod \"node-ca-nqv9s\" (UID: \"a2127c7c-06fc-4849-ab98-2975c0862540\") " pod="openshift-image-registry/node-ca-nqv9s" Jul 29 00:54:11.266559 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265854 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-host-run-k8s-cni-cncf-io\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.266559 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265865 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-host-cni-bin\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.266559 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:11.265841 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 00:54:11.266559 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265884 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/03f35a91-81af-4c82-976b-50ad3b9d4ea6-kubelet-config\") pod \"global-pull-secret-syncer-lf9gq\" (UID: \"03f35a91-81af-4c82-976b-50ad3b9d4ea6\") " pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:11.266559 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265918 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-host-cni-bin\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.266559 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265928 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a557028-eba5-431a-8690-dd95f88e82df-ovn-node-metrics-cert\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.266559 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265866 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f974fab-a19b-419d-bc9f-ff551c5d928f-system-cni-dir\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.266559 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265958 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b5qg\" (UniqueName: \"kubernetes.io/projected/78bcb31a-2890-4527-b8f0-73cabeaecd3e-kube-api-access-7b5qg\") pod \"network-metrics-daemon-jx7tt\" (UID: \"78bcb31a-2890-4527-b8f0-73cabeaecd3e\") " pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:11.266559 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265982 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2127c7c-06fc-4849-ab98-2975c0862540-host\") pod \"node-ca-nqv9s\" (UID: \"a2127c7c-06fc-4849-ab98-2975c0862540\") " pod="openshift-image-registry/node-ca-nqv9s" Jul 29 00:54:11.266559 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266003 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-host-run-netns\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.266559 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266034 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-log-socket\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.266559 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.265866 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-host-kubelet\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.266559 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:11.266061 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret podName:03f35a91-81af-4c82-976b-50ad3b9d4ea6 nodeName:}" failed. No retries permitted until 2026-07-29 00:54:11.766040935 +0000 UTC m=+2.036685241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret") pod "global-pull-secret-syncer-lf9gq" (UID: "03f35a91-81af-4c82-976b-50ad3b9d4ea6") : object "kube-system"/"original-pull-secret" not registered Jul 29 00:54:11.266559 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266076 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-log-socket\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.266559 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266097 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs\") pod \"network-metrics-daemon-jx7tt\" (UID: \"78bcb31a-2890-4527-b8f0-73cabeaecd3e\") " pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:11.267396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266127 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-multus-cni-dir\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.267396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266152 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-os-release\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.267396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266161 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a2127c7c-06fc-4849-ab98-2975c0862540-host\") pod \"node-ca-nqv9s\" (UID: \"a2127c7c-06fc-4849-ab98-2975c0862540\") " pod="openshift-image-registry/node-ca-nqv9s" Jul 29 00:54:11.267396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266176 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-run-openvswitch\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.267396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266120 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-host-run-netns\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.267396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266201 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-node-log\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.267396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266238 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-node-log\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.267396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266237 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f974fab-a19b-419d-bc9f-ff551c5d928f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.267396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266281 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-host-var-lib-kubelet\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.267396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266297 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-multus-cni-dir\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.267396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266307 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-run-systemd\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.267396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266335 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-os-release\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.267396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266355 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-var-lib-openvswitch\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.267396 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:11.266383 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 00:54:11.267396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266383 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.267396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266415 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-run-openvswitch\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.267396 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266417 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-host-var-lib-cni-bin\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.268042 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:11.266446 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs podName:78bcb31a-2890-4527-b8f0-73cabeaecd3e nodeName:}" failed. No retries permitted until 2026-07-29 00:54:11.766426868 +0000 UTC m=+2.037071164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs") pod "network-metrics-daemon-jx7tt" (UID: "78bcb31a-2890-4527-b8f0-73cabeaecd3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 00:54:11.268042 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266453 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-host-var-lib-cni-bin\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.268042 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266479 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-etc-kubernetes\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.268042 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266503 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-run-systemd\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.268042 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266542 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-hostroot\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.268042 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266551 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-etc-kubernetes\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.268042 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266386 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-host-var-lib-kubelet\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.268042 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266581 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-var-lib-openvswitch\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.268042 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266507 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-hostroot\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.268042 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266595 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a557028-eba5-431a-8690-dd95f88e82df-ovnkube-script-lib\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.268042 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266609 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.268042 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266619 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a557028-eba5-431a-8690-dd95f88e82df-ovnkube-config\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.268042 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266646 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f974fab-a19b-419d-bc9f-ff551c5d928f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.268042 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266670 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47a76b9a-5cc4-4739-8ab3-2745dab6d404-host-slash\") pod \"iptables-alerter-lsz6j\" (UID: \"47a76b9a-5cc4-4739-8ab3-2745dab6d404\") " pod="openshift-network-operator/iptables-alerter-lsz6j" Jul 29 00:54:11.268042 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266698 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07544af8-8a7f-4ada-add4-28bc48719e32-cni-binary-copy\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.268042 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266725 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03f35a91-81af-4c82-976b-50ad3b9d4ea6-dbus\") pod \"global-pull-secret-syncer-lf9gq\" (UID: \"03f35a91-81af-4c82-976b-50ad3b9d4ea6\") " pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:11.268042 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266749 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-host-slash\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.268589 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266773 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-etc-openvswitch\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.268589 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266797 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh7nc\" (UniqueName: \"kubernetes.io/projected/47a76b9a-5cc4-4739-8ab3-2745dab6d404-kube-api-access-vh7nc\") pod \"iptables-alerter-lsz6j\" (UID: \"47a76b9a-5cc4-4739-8ab3-2745dab6d404\") " pod="openshift-network-operator/iptables-alerter-lsz6j" Jul 29 00:54:11.268589 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266825 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd7x9\" (UniqueName: \"kubernetes.io/projected/07544af8-8a7f-4ada-add4-28bc48719e32-kube-api-access-dd7x9\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.268589 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266852 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-host-run-ovn-kubernetes\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.268589 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266878 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-host-cni-netd\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.268589 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266915 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f974fab-a19b-419d-bc9f-ff551c5d928f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.268589 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266925 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-host-slash\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.268589 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266920 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f974fab-a19b-419d-bc9f-ff551c5d928f-cnibin\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.268589 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.266964 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-etc-openvswitch\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.268589 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.267115 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f974fab-a19b-419d-bc9f-ff551c5d928f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.268589 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.267137 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47a76b9a-5cc4-4739-8ab3-2745dab6d404-host-slash\") pod \"iptables-alerter-lsz6j\" (UID: \"47a76b9a-5cc4-4739-8ab3-2745dab6d404\") " pod="openshift-network-operator/iptables-alerter-lsz6j" Jul 29 00:54:11.268589 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.267147 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a557028-eba5-431a-8690-dd95f88e82df-ovnkube-config\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.268589 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.267171 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-multus-socket-dir-parent\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.268589 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.267201 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-host-run-netns\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.268589 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.267236 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1f974fab-a19b-419d-bc9f-ff551c5d928f-os-release\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.268589 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.267174 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f974fab-a19b-419d-bc9f-ff551c5d928f-cnibin\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.268589 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.267286 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f974fab-a19b-419d-bc9f-ff551c5d928f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.269171 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.267369 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-host-cni-netd\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.269171 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.267521 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-multus-socket-dir-parent\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.269171 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.267531 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/03f35a91-81af-4c82-976b-50ad3b9d4ea6-dbus\") pod \"global-pull-secret-syncer-lf9gq\" (UID: \"03f35a91-81af-4c82-976b-50ad3b9d4ea6\") " pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:11.269171 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.267565 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1f974fab-a19b-419d-bc9f-ff551c5d928f-os-release\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.269171 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.267571 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a557028-eba5-431a-8690-dd95f88e82df-host-run-ovn-kubernetes\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.269171 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.267600 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07544af8-8a7f-4ada-add4-28bc48719e32-host-run-netns\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.269171 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.267616 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07544af8-8a7f-4ada-add4-28bc48719e32-cni-binary-copy\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.269171 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.267768 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f974fab-a19b-419d-bc9f-ff551c5d928f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.269171 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.268518 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a557028-eba5-431a-8690-dd95f88e82df-ovn-node-metrics-cert\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.272730 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.272705 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwzct\" (UniqueName: \"kubernetes.io/projected/a2127c7c-06fc-4849-ab98-2975c0862540-kube-api-access-mwzct\") pod \"node-ca-nqv9s\" (UID: \"a2127c7c-06fc-4849-ab98-2975c0862540\") " pod="openshift-image-registry/node-ca-nqv9s" Jul 29 00:54:11.272876 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.272858 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4ctw\" (UniqueName: \"kubernetes.io/projected/1f974fab-a19b-419d-bc9f-ff551c5d928f-kube-api-access-l4ctw\") pod \"multus-additional-cni-plugins-d7g65\" (UID: \"1f974fab-a19b-419d-bc9f-ff551c5d928f\") " pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.276783 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.276757 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmmk7\" (UniqueName: \"kubernetes.io/projected/8a557028-eba5-431a-8690-dd95f88e82df-kube-api-access-wmmk7\") pod \"ovnkube-node-h8nv4\" (UID: \"8a557028-eba5-431a-8690-dd95f88e82df\") " pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.277087 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.277070 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh7nc\" (UniqueName: \"kubernetes.io/projected/47a76b9a-5cc4-4739-8ab3-2745dab6d404-kube-api-access-vh7nc\") pod \"iptables-alerter-lsz6j\" (UID: \"47a76b9a-5cc4-4739-8ab3-2745dab6d404\") " pod="openshift-network-operator/iptables-alerter-lsz6j" Jul 29 00:54:11.277149 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.277128 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b5qg\" (UniqueName: \"kubernetes.io/projected/78bcb31a-2890-4527-b8f0-73cabeaecd3e-kube-api-access-7b5qg\") pod \"network-metrics-daemon-jx7tt\" (UID: \"78bcb31a-2890-4527-b8f0-73cabeaecd3e\") " pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:11.277930 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.277912 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd7x9\" (UniqueName: \"kubernetes.io/projected/07544af8-8a7f-4ada-add4-28bc48719e32-kube-api-access-dd7x9\") pod \"multus-2mrv7\" (UID: \"07544af8-8a7f-4ada-add4-28bc48719e32\") " pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.286976 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:11.286918 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc032dfa2331a0d7e82f450dc05e5aa6c.slice/crio-5081a3be53cd95199b4e215498c7a03e2b253984716e843a874a948105305134 WatchSource:0}: Error finding container 5081a3be53cd95199b4e215498c7a03e2b253984716e843a874a948105305134: Status 404 returned error can't find the container with id 5081a3be53cd95199b4e215498c7a03e2b253984716e843a874a948105305134 Jul 29 00:54:11.289643 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:11.289619 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b4121d255f123d5bc2e0fe24413a8b.slice/crio-3d47f84580f8f2ede8c14f5a8b7fff0a50f318f5ee89f0a65c5431c5965f5703 WatchSource:0}: Error finding container 3d47f84580f8f2ede8c14f5a8b7fff0a50f318f5ee89f0a65c5431c5965f5703: Status 404 returned error can't find the container with id 3d47f84580f8f2ede8c14f5a8b7fff0a50f318f5ee89f0a65c5431c5965f5703 Jul 29 00:54:11.292748 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.292729 2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 00:54:11.340691 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.340640 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-136.ec2.internal" event={"ID":"c032dfa2331a0d7e82f450dc05e5aa6c","Type":"ContainerStarted","Data":"5081a3be53cd95199b4e215498c7a03e2b253984716e843a874a948105305134"} Jul 29 00:54:11.341682 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.341659 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-136.ec2.internal" event={"ID":"e1b4121d255f123d5bc2e0fe24413a8b","Type":"ContainerStarted","Data":"3d47f84580f8f2ede8c14f5a8b7fff0a50f318f5ee89f0a65c5431c5965f5703"} Jul 29 00:54:11.372377 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.372359 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" Jul 29 00:54:11.380436 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:11.380413 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79868cd_1ac6_4262_ab1f_687d466f8b4d.slice/crio-de5120521dd09271d5a2df78d65e30fb340ed5f0883157d7c81954f53cb255c3 WatchSource:0}: Error finding container de5120521dd09271d5a2df78d65e30fb340ed5f0883157d7c81954f53cb255c3: Status 404 returned error can't find the container with id de5120521dd09271d5a2df78d65e30fb340ed5f0883157d7c81954f53cb255c3 Jul 29 00:54:11.391065 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.391049 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j7xsc" Jul 29 00:54:11.398506 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:11.398486 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b304c44_8f9d_4830_a2e7_d5559eeb879f.slice/crio-af1fdbcb6ab59b3f3500534ccbb47497153cce57f48237f5669f0f3ac14a7016 WatchSource:0}: Error finding container af1fdbcb6ab59b3f3500534ccbb47497153cce57f48237f5669f0f3ac14a7016: Status 404 returned error can't find the container with id af1fdbcb6ab59b3f3500534ccbb47497153cce57f48237f5669f0f3ac14a7016 Jul 29 00:54:11.399252 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.399234 2543 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 00:54:11.405370 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.405354 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jh44t" Jul 29 00:54:11.411898 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:11.411866 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a3edfc_262f_4bb8_96fb_4b3b2542d9af.slice/crio-564849a92256b6bfe03f63adc33a04e4ad70d825df80d87659418ba3b405afcc WatchSource:0}: Error finding container 564849a92256b6bfe03f63adc33a04e4ad70d825df80d87659418ba3b405afcc: Status 404 returned error can't find the container with id 564849a92256b6bfe03f63adc33a04e4ad70d825df80d87659418ba3b405afcc Jul 29 00:54:11.420860 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.420835 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-thq5l" Jul 29 00:54:11.427558 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:11.427535 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5719a514_b8de_4661_ab2a_fe0587726ae8.slice/crio-feba865f9a913a918311ff2b36074beaa0a83548ffa4245dbf843b83199b1e8e WatchSource:0}: Error finding container feba865f9a913a918311ff2b36074beaa0a83548ffa4245dbf843b83199b1e8e: Status 404 returned error can't find the container with id feba865f9a913a918311ff2b36074beaa0a83548ffa4245dbf843b83199b1e8e Jul 29 00:54:11.436558 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.436544 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nqv9s" Jul 29 00:54:11.443478 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:11.443447 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2127c7c_06fc_4849_ab98_2975c0862540.slice/crio-ea944a5212e0556ba2b75c7c7aff89651f59dccfe2f45e56283d69e98ab1bf95 WatchSource:0}: Error finding container ea944a5212e0556ba2b75c7c7aff89651f59dccfe2f45e56283d69e98ab1bf95: Status 404 returned error can't find the container with id ea944a5212e0556ba2b75c7c7aff89651f59dccfe2f45e56283d69e98ab1bf95 Jul 29 00:54:11.452880 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.452861 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:11.459695 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:11.459674 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a557028_eba5_431a_8690_dd95f88e82df.slice/crio-eb694a0216333047c2ad0e9d5a04fa71df14fa6307c2da9eb2fc5ed7700cbca4 WatchSource:0}: Error finding container eb694a0216333047c2ad0e9d5a04fa71df14fa6307c2da9eb2fc5ed7700cbca4: Status 404 returned error can't find the container with id eb694a0216333047c2ad0e9d5a04fa71df14fa6307c2da9eb2fc5ed7700cbca4 Jul 29 00:54:11.465599 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.465582 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d7g65" Jul 29 00:54:11.472170 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:11.472148 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f974fab_a19b_419d_bc9f_ff551c5d928f.slice/crio-a281408647cd44fd0c63f540302cacba9d1fc061304eb3704b31fc3974fd3b00 WatchSource:0}: Error finding container a281408647cd44fd0c63f540302cacba9d1fc061304eb3704b31fc3974fd3b00: Status 404 returned error can't find the container with id a281408647cd44fd0c63f540302cacba9d1fc061304eb3704b31fc3974fd3b00 Jul 29 00:54:11.491886 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.491863 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2mrv7" Jul 29 00:54:11.498572 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:11.498550 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07544af8_8a7f_4ada_add4_28bc48719e32.slice/crio-f3908701216959af5836b53d1bbf64bed16ed305fd65273c5b7c12d656daa28b WatchSource:0}: Error finding container f3908701216959af5836b53d1bbf64bed16ed305fd65273c5b7c12d656daa28b: Status 404 returned error can't find the container with id f3908701216959af5836b53d1bbf64bed16ed305fd65273c5b7c12d656daa28b Jul 29 00:54:11.502104 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.502084 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lsz6j" Jul 29 00:54:11.509124 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:11.509101 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a76b9a_5cc4_4739_8ab3_2745dab6d404.slice/crio-e42581f4eb43d6e3a87a702508aa71f5d00b6973ff3dcbceac9b9c82dd1ef17c WatchSource:0}: Error finding container e42581f4eb43d6e3a87a702508aa71f5d00b6973ff3dcbceac9b9c82dd1ef17c: Status 404 returned error can't find the container with id e42581f4eb43d6e3a87a702508aa71f5d00b6973ff3dcbceac9b9c82dd1ef17c Jul 29 00:54:11.770700 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.770610 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret\") pod \"global-pull-secret-syncer-lf9gq\" (UID: \"03f35a91-81af-4c82-976b-50ad3b9d4ea6\") " pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:11.770700 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.770658 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jspd\" (UniqueName: \"kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd\") pod \"network-check-target-g87tb\" (UID: \"926d37fa-f392-43d9-b446-160238607eff\") " pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:11.770942 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:11.770776 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 00:54:11.770942 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:11.770795 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 00:54:11.770942 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:11.770796 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 00:54:11.770942 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:11.770801 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs\") pod \"network-metrics-daemon-jx7tt\" (UID: \"78bcb31a-2890-4527-b8f0-73cabeaecd3e\") " pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:11.770942 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:11.770808 2543 projected.go:196] Error preparing data for projected volume kube-api-access-5jspd for pod openshift-network-diagnostics/network-check-target-g87tb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 00:54:11.770942 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:11.770890 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 00:54:11.770942 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:11.770898 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret podName:03f35a91-81af-4c82-976b-50ad3b9d4ea6 nodeName:}" failed. No retries permitted until 2026-07-29 00:54:12.770876682 +0000 UTC m=+3.041520986 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret") pod "global-pull-secret-syncer-lf9gq" (UID: "03f35a91-81af-4c82-976b-50ad3b9d4ea6") : object "kube-system"/"original-pull-secret" not registered Jul 29 00:54:11.771356 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:11.770957 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd podName:926d37fa-f392-43d9-b446-160238607eff nodeName:}" failed. No retries permitted until 2026-07-29 00:54:12.770941675 +0000 UTC m=+3.041585965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5jspd" (UniqueName: "kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd") pod "network-check-target-g87tb" (UID: "926d37fa-f392-43d9-b446-160238607eff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 00:54:11.771356 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:11.770973 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs podName:78bcb31a-2890-4527-b8f0-73cabeaecd3e nodeName:}" failed. No retries permitted until 2026-07-29 00:54:12.770965513 +0000 UTC m=+3.041609800 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs") pod "network-metrics-daemon-jx7tt" (UID: "78bcb31a-2890-4527-b8f0-73cabeaecd3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 00:54:12.008016 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:12.007980 2543 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 00:54:12.094504 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:12.094383 2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 00:49:11 +0000 UTC" deadline="2028-04-29 03:20:45.978955644 +0000 UTC" Jul 29 00:54:12.094504 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:12.094452 2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15362h26m33.884508395s" Jul 29 00:54:12.341444 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:12.340552 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:12.341444 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:12.340700 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jx7tt" podUID="78bcb31a-2890-4527-b8f0-73cabeaecd3e" Jul 29 00:54:12.341444 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:12.341136 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:12.341444 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:12.341233 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g87tb" podUID="926d37fa-f392-43d9-b446-160238607eff" Jul 29 00:54:12.341444 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:12.341302 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:12.341444 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:12.341377 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lf9gq" podUID="03f35a91-81af-4c82-976b-50ad3b9d4ea6" Jul 29 00:54:12.358866 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:12.358785 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7g65" event={"ID":"1f974fab-a19b-419d-bc9f-ff551c5d928f","Type":"ContainerStarted","Data":"a281408647cd44fd0c63f540302cacba9d1fc061304eb3704b31fc3974fd3b00"} Jul 29 00:54:12.378745 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:12.378709 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-thq5l" event={"ID":"5719a514-b8de-4661-ab2a-fe0587726ae8","Type":"ContainerStarted","Data":"feba865f9a913a918311ff2b36074beaa0a83548ffa4245dbf843b83199b1e8e"} Jul 29 00:54:12.396756 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:12.396726 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jh44t" event={"ID":"11a3edfc-262f-4bb8-96fb-4b3b2542d9af","Type":"ContainerStarted","Data":"564849a92256b6bfe03f63adc33a04e4ad70d825df80d87659418ba3b405afcc"} Jul 29 00:54:12.451014 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:12.450972 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" event={"ID":"e79868cd-1ac6-4262-ab1f-687d466f8b4d","Type":"ContainerStarted","Data":"de5120521dd09271d5a2df78d65e30fb340ed5f0883157d7c81954f53cb255c3"} Jul 29 00:54:12.459355 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:12.457222 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lsz6j" event={"ID":"47a76b9a-5cc4-4739-8ab3-2745dab6d404","Type":"ContainerStarted","Data":"e42581f4eb43d6e3a87a702508aa71f5d00b6973ff3dcbceac9b9c82dd1ef17c"} Jul 29 00:54:12.471775 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:12.471745 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2mrv7" event={"ID":"07544af8-8a7f-4ada-add4-28bc48719e32","Type":"ContainerStarted","Data":"f3908701216959af5836b53d1bbf64bed16ed305fd65273c5b7c12d656daa28b"} Jul 29 00:54:12.486354 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:12.486234 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" event={"ID":"8a557028-eba5-431a-8690-dd95f88e82df","Type":"ContainerStarted","Data":"eb694a0216333047c2ad0e9d5a04fa71df14fa6307c2da9eb2fc5ed7700cbca4"} Jul 29 00:54:12.496249 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:12.495913 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nqv9s" event={"ID":"a2127c7c-06fc-4849-ab98-2975c0862540","Type":"ContainerStarted","Data":"ea944a5212e0556ba2b75c7c7aff89651f59dccfe2f45e56283d69e98ab1bf95"} Jul 29 00:54:12.504216 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:12.504191 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j7xsc" event={"ID":"0b304c44-8f9d-4830-a2e7-d5559eeb879f","Type":"ContainerStarted","Data":"af1fdbcb6ab59b3f3500534ccbb47497153cce57f48237f5669f0f3ac14a7016"} Jul 29 00:54:12.783298 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:12.782297 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret\") pod \"global-pull-secret-syncer-lf9gq\" (UID: \"03f35a91-81af-4c82-976b-50ad3b9d4ea6\") " pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:12.783298 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:12.782343 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jspd\" (UniqueName: \"kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd\") pod \"network-check-target-g87tb\" (UID: \"926d37fa-f392-43d9-b446-160238607eff\") " pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:12.783298 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:12.782379 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs\") pod \"network-metrics-daemon-jx7tt\" (UID: \"78bcb31a-2890-4527-b8f0-73cabeaecd3e\") " pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:12.783298 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:12.782524 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 00:54:12.783298 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:12.782616 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs podName:78bcb31a-2890-4527-b8f0-73cabeaecd3e nodeName:}" failed. No retries permitted until 2026-07-29 00:54:14.782595151 +0000 UTC m=+5.053239439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs") pod "network-metrics-daemon-jx7tt" (UID: "78bcb31a-2890-4527-b8f0-73cabeaecd3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 00:54:12.783298 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:12.782713 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 00:54:12.783298 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:12.782754 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret podName:03f35a91-81af-4c82-976b-50ad3b9d4ea6 nodeName:}" failed. No retries permitted until 2026-07-29 00:54:14.782739808 +0000 UTC m=+5.053384099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret") pod "global-pull-secret-syncer-lf9gq" (UID: "03f35a91-81af-4c82-976b-50ad3b9d4ea6") : object "kube-system"/"original-pull-secret" not registered Jul 29 00:54:12.783298 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:12.782846 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 00:54:12.783298 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:12.782861 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 00:54:12.783298 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:12.782889 2543 projected.go:196] Error preparing data for projected volume kube-api-access-5jspd for pod openshift-network-diagnostics/network-check-target-g87tb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 00:54:12.783298 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:12.782927 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd podName:926d37fa-f392-43d9-b446-160238607eff nodeName:}" failed. No retries permitted until 2026-07-29 00:54:14.782913311 +0000 UTC m=+5.053557611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5jspd" (UniqueName: "kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd") pod "network-check-target-g87tb" (UID: "926d37fa-f392-43d9-b446-160238607eff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 00:54:13.095778 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:13.095685 2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 00:49:11 +0000 UTC" deadline="2028-03-23 01:19:46.79345111 +0000 UTC" Jul 29 00:54:13.095778 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:13.095728 2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14472h25m33.69772795s" Jul 29 00:54:13.214308 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:13.214142 2543 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 00:54:14.340230 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:14.340125 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:14.340704 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:14.340258 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lf9gq" podUID="03f35a91-81af-4c82-976b-50ad3b9d4ea6" Jul 29 00:54:14.340704 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:14.340672 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:14.340830 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:14.340771 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g87tb" podUID="926d37fa-f392-43d9-b446-160238607eff" Jul 29 00:54:14.340830 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:14.340824 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:14.340967 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:14.340935 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jx7tt" podUID="78bcb31a-2890-4527-b8f0-73cabeaecd3e" Jul 29 00:54:14.801300 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:14.801206 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret\") pod \"global-pull-secret-syncer-lf9gq\" (UID: \"03f35a91-81af-4c82-976b-50ad3b9d4ea6\") " pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:14.801300 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:14.801263 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jspd\" (UniqueName: \"kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd\") pod \"network-check-target-g87tb\" (UID: \"926d37fa-f392-43d9-b446-160238607eff\") " pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:14.801546 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:14.801299 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs\") pod \"network-metrics-daemon-jx7tt\" (UID: \"78bcb31a-2890-4527-b8f0-73cabeaecd3e\") " pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:14.801546 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:14.801449 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 00:54:14.801546 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:14.801517 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs podName:78bcb31a-2890-4527-b8f0-73cabeaecd3e nodeName:}" failed. No retries permitted until 2026-07-29 00:54:18.801497865 +0000 UTC m=+9.072142168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs") pod "network-metrics-daemon-jx7tt" (UID: "78bcb31a-2890-4527-b8f0-73cabeaecd3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 00:54:14.801970 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:14.801948 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 00:54:14.802086 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:14.802010 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret podName:03f35a91-81af-4c82-976b-50ad3b9d4ea6 nodeName:}" failed. No retries permitted until 2026-07-29 00:54:18.801992936 +0000 UTC m=+9.072637238 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret") pod "global-pull-secret-syncer-lf9gq" (UID: "03f35a91-81af-4c82-976b-50ad3b9d4ea6") : object "kube-system"/"original-pull-secret" not registered Jul 29 00:54:14.802154 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:14.802091 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 00:54:14.802154 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:14.802109 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 00:54:14.802154 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:14.802122 2543 projected.go:196] Error preparing data for projected volume kube-api-access-5jspd for pod openshift-network-diagnostics/network-check-target-g87tb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 00:54:14.802301 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:14.802159 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd podName:926d37fa-f392-43d9-b446-160238607eff nodeName:}" failed. No retries permitted until 2026-07-29 00:54:18.802147241 +0000 UTC m=+9.072791545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5jspd" (UniqueName: "kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd") pod "network-check-target-g87tb" (UID: "926d37fa-f392-43d9-b446-160238607eff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 00:54:16.338185 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:16.338153 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:16.338185 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:16.338186 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:16.338722 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:16.338153 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:16.338722 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:16.338347 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g87tb" podUID="926d37fa-f392-43d9-b446-160238607eff" Jul 29 00:54:16.338722 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:16.338417 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jx7tt" podUID="78bcb31a-2890-4527-b8f0-73cabeaecd3e" Jul 29 00:54:16.338722 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:16.338496 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lf9gq" podUID="03f35a91-81af-4c82-976b-50ad3b9d4ea6" Jul 29 00:54:18.337286 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:18.337246 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:18.337286 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:18.337285 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:18.337738 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:18.337382 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g87tb" podUID="926d37fa-f392-43d9-b446-160238607eff" Jul 29 00:54:18.337738 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:18.337665 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:18.337871 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:18.337783 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jx7tt" podUID="78bcb31a-2890-4527-b8f0-73cabeaecd3e" Jul 29 00:54:18.337928 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:18.337895 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lf9gq" podUID="03f35a91-81af-4c82-976b-50ad3b9d4ea6" Jul 29 00:54:18.833519 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:18.832456 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jspd\" (UniqueName: \"kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd\") pod \"network-check-target-g87tb\" (UID: \"926d37fa-f392-43d9-b446-160238607eff\") " pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:18.833519 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:18.832515 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs\") pod \"network-metrics-daemon-jx7tt\" (UID: \"78bcb31a-2890-4527-b8f0-73cabeaecd3e\") " pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:18.833519 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:18.832598 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret\") pod \"global-pull-secret-syncer-lf9gq\" (UID: \"03f35a91-81af-4c82-976b-50ad3b9d4ea6\") " pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:18.833519 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:18.832733 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 00:54:18.833519 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:18.832797 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret podName:03f35a91-81af-4c82-976b-50ad3b9d4ea6 nodeName:}" failed. No retries permitted until 2026-07-29 00:54:26.832778052 +0000 UTC m=+17.103422341 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret") pod "global-pull-secret-syncer-lf9gq" (UID: "03f35a91-81af-4c82-976b-50ad3b9d4ea6") : object "kube-system"/"original-pull-secret" not registered Jul 29 00:54:18.833519 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:18.833227 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 00:54:18.833519 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:18.833250 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 00:54:18.833519 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:18.833263 2543 projected.go:196] Error preparing data for projected volume kube-api-access-5jspd for pod openshift-network-diagnostics/network-check-target-g87tb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 00:54:18.833519 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:18.833319 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd podName:926d37fa-f392-43d9-b446-160238607eff nodeName:}" failed. No retries permitted until 2026-07-29 00:54:26.833303398 +0000 UTC m=+17.103947696 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5jspd" (UniqueName: "kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd") pod "network-check-target-g87tb" (UID: "926d37fa-f392-43d9-b446-160238607eff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 00:54:18.833519 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:18.833390 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 00:54:18.833519 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:18.833426 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs podName:78bcb31a-2890-4527-b8f0-73cabeaecd3e nodeName:}" failed. No retries permitted until 2026-07-29 00:54:26.833413534 +0000 UTC m=+17.104057820 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs") pod "network-metrics-daemon-jx7tt" (UID: "78bcb31a-2890-4527-b8f0-73cabeaecd3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 00:54:20.338839 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:20.338784 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:20.338839 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:20.338830 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:20.339382 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:20.338783 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:20.339382 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:20.338939 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jx7tt" podUID="78bcb31a-2890-4527-b8f0-73cabeaecd3e" Jul 29 00:54:20.339382 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:20.339031 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g87tb" podUID="926d37fa-f392-43d9-b446-160238607eff" Jul 29 00:54:20.339382 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:20.339119 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lf9gq" podUID="03f35a91-81af-4c82-976b-50ad3b9d4ea6" Jul 29 00:54:22.337871 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:22.337808 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:22.338310 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:22.337806 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:22.338310 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:22.337836 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:22.338310 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:22.337946 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g87tb" podUID="926d37fa-f392-43d9-b446-160238607eff" Jul 29 00:54:22.338310 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:22.338113 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jx7tt" podUID="78bcb31a-2890-4527-b8f0-73cabeaecd3e" Jul 29 00:54:22.338310 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:22.338199 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lf9gq" podUID="03f35a91-81af-4c82-976b-50ad3b9d4ea6" Jul 29 00:54:24.337571 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:24.337536 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:24.338040 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:24.337538 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:24.338040 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:24.337652 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g87tb" podUID="926d37fa-f392-43d9-b446-160238607eff" Jul 29 00:54:24.338040 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:24.337741 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lf9gq" podUID="03f35a91-81af-4c82-976b-50ad3b9d4ea6" Jul 29 00:54:24.338040 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:24.337545 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:24.338040 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:24.337885 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jx7tt" podUID="78bcb31a-2890-4527-b8f0-73cabeaecd3e" Jul 29 00:54:26.337175 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:26.337134 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:26.337175 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:26.337164 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:26.337714 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:26.337287 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lf9gq" podUID="03f35a91-81af-4c82-976b-50ad3b9d4ea6" Jul 29 00:54:26.337714 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:26.337340 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:26.337714 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:26.337467 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g87tb" podUID="926d37fa-f392-43d9-b446-160238607eff" Jul 29 00:54:26.337714 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:26.337556 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jx7tt" podUID="78bcb31a-2890-4527-b8f0-73cabeaecd3e" Jul 29 00:54:26.894461 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:26.894425 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret\") pod \"global-pull-secret-syncer-lf9gq\" (UID: \"03f35a91-81af-4c82-976b-50ad3b9d4ea6\") " pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:26.894461 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:26.894464 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jspd\" (UniqueName: \"kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd\") pod \"network-check-target-g87tb\" (UID: \"926d37fa-f392-43d9-b446-160238607eff\") " pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:26.894703 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:26.894497 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs\") pod \"network-metrics-daemon-jx7tt\" (UID: \"78bcb31a-2890-4527-b8f0-73cabeaecd3e\") " pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:26.894703 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:26.894594 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 00:54:26.894703 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:26.894611 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 00:54:26.894703 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:26.894637 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 00:54:26.894703 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:26.894651 2543 projected.go:196] Error preparing data for projected volume kube-api-access-5jspd for pod openshift-network-diagnostics/network-check-target-g87tb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 00:54:26.894703 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:26.894660 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret podName:03f35a91-81af-4c82-976b-50ad3b9d4ea6 nodeName:}" failed. No retries permitted until 2026-07-29 00:54:42.894643976 +0000 UTC m=+33.165288261 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret") pod "global-pull-secret-syncer-lf9gq" (UID: "03f35a91-81af-4c82-976b-50ad3b9d4ea6") : object "kube-system"/"original-pull-secret" not registered Jul 29 00:54:26.894703 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:26.894603 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 00:54:26.894703 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:26.894699 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd podName:926d37fa-f392-43d9-b446-160238607eff nodeName:}" failed. No retries permitted until 2026-07-29 00:54:42.894683808 +0000 UTC m=+33.165328111 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5jspd" (UniqueName: "kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd") pod "network-check-target-g87tb" (UID: "926d37fa-f392-43d9-b446-160238607eff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 00:54:26.895021 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:26.894730 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs podName:78bcb31a-2890-4527-b8f0-73cabeaecd3e nodeName:}" failed. No retries permitted until 2026-07-29 00:54:42.894714332 +0000 UTC m=+33.165358639 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs") pod "network-metrics-daemon-jx7tt" (UID: "78bcb31a-2890-4527-b8f0-73cabeaecd3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 00:54:28.337102 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:28.337065 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:28.337638 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:28.337065 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:28.337638 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:28.337182 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g87tb" podUID="926d37fa-f392-43d9-b446-160238607eff" Jul 29 00:54:28.337638 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:28.337223 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:28.337638 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:28.337323 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lf9gq" podUID="03f35a91-81af-4c82-976b-50ad3b9d4ea6" Jul 29 00:54:28.337638 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:28.337449 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jx7tt" podUID="78bcb31a-2890-4527-b8f0-73cabeaecd3e" Jul 29 00:54:30.338632 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:30.338422 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:30.339096 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:30.338512 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:30.339096 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:30.338724 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g87tb" podUID="926d37fa-f392-43d9-b446-160238607eff" Jul 29 00:54:30.339096 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:30.338568 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:30.339096 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:30.338849 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jx7tt" podUID="78bcb31a-2890-4527-b8f0-73cabeaecd3e" Jul 29 00:54:30.339096 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:30.338940 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lf9gq" podUID="03f35a91-81af-4c82-976b-50ad3b9d4ea6" Jul 29 00:54:32.337965 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:32.337943 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:32.338292 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:32.337986 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:32.338292 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:32.338078 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lf9gq" podUID="03f35a91-81af-4c82-976b-50ad3b9d4ea6" Jul 29 00:54:32.338392 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:32.338359 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jx7tt" podUID="78bcb31a-2890-4527-b8f0-73cabeaecd3e" Jul 29 00:54:32.338448 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:32.338404 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:32.338503 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:32.338482 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g87tb" podUID="926d37fa-f392-43d9-b446-160238607eff" Jul 29 00:54:32.555522 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:32.555494 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jh44t" event={"ID":"11a3edfc-262f-4bb8-96fb-4b3b2542d9af","Type":"ContainerStarted","Data":"b7064f6b1fe3c9e55e3e5910ec6ba065f35a1729a477d79aba278b2ca32be092"} Jul 29 00:54:32.557045 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:32.557022 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2mrv7" event={"ID":"07544af8-8a7f-4ada-add4-28bc48719e32","Type":"ContainerStarted","Data":"31c6348f778e8aaafd3fe06d62ee471ef24e9904a7a1eb9c69c37876db4d25b7"} Jul 29 00:54:32.559095 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:32.559077 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" event={"ID":"8a557028-eba5-431a-8690-dd95f88e82df","Type":"ContainerStarted","Data":"db125adc8ca9f31a398fa3c3e1d779aae4936395abcb6145f64adc4d85a0fed2"} Jul 29 00:54:32.559182 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:32.559153 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" event={"ID":"8a557028-eba5-431a-8690-dd95f88e82df","Type":"ContainerStarted","Data":"23b551a50446699de421c47ba6c7cf8e8a879793709c27a723558b371d5cba2c"} Jul 29 00:54:32.559245 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:32.559203 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" event={"ID":"8a557028-eba5-431a-8690-dd95f88e82df","Type":"ContainerStarted","Data":"3bf6c7ef45d46de4c98615707fb79e50729819465670416cbaa234314fff1040"} Jul 29 00:54:32.560402 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:32.560379 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-136.ec2.internal" event={"ID":"e1b4121d255f123d5bc2e0fe24413a8b","Type":"ContainerStarted","Data":"e07bcae64da8dd5c294d938273b8373f516067fd7f46a19bfc02a1cb2c6c3656"} Jul 29 00:54:32.571190 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:32.571142 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jh44t" podStartSLOduration=2.201971689 podStartE2EDuration="22.57112377s" totalImagesPullingTime="20.369152081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:54:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:54:32.570771136 +0000 UTC m=+22.841415481" watchObservedRunningTime="2026-07-29 00:54:32.57112377 +0000 UTC m=+22.841768077" Jul 29 00:54:32.585118 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:32.585083 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2mrv7" podStartSLOduration=2.271195741 podStartE2EDuration="22.585071956s" totalImagesPullingTime="20.313876215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:54:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:54:32.584990942 +0000 UTC m=+22.855635249" watchObservedRunningTime="2026-07-29 00:54:32.585071956 +0000 UTC m=+22.855716263" Jul 29 00:54:33.564615 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:33.564502 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j7xsc" event={"ID":"0b304c44-8f9d-4830-a2e7-d5559eeb879f","Type":"ContainerStarted","Data":"8ab720cc7de91238b80d3a9c7f0e403c487e79367f60ec1a2ff7820800b8e6cf"} Jul 29 00:54:33.566201 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:33.566172 2543 generic.go:356] "Generic (PLEG): container finished" podID="c032dfa2331a0d7e82f450dc05e5aa6c" containerID="a3b536bd4fb201404d2b676209ebe9ba99e6c471a5f34257c20cd85cec3146a4" exitCode=0 Jul 29 00:54:33.566323 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:33.566260 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-136.ec2.internal" event={"ID":"c032dfa2331a0d7e82f450dc05e5aa6c","Type":"ContainerDied","Data":"a3b536bd4fb201404d2b676209ebe9ba99e6c471a5f34257c20cd85cec3146a4"} Jul 29 00:54:33.569166 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:33.569137 2543 generic.go:356] "Generic (PLEG): container finished" podID="1f974fab-a19b-419d-bc9f-ff551c5d928f" containerID="25a294cfbb78c6b8e533f68210e6bc3b0091c38a79eca9f90137ccc37be57324" exitCode=0 Jul 29 00:54:33.569280 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:33.569220 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7g65" event={"ID":"1f974fab-a19b-419d-bc9f-ff551c5d928f","Type":"ContainerDied","Data":"25a294cfbb78c6b8e533f68210e6bc3b0091c38a79eca9f90137ccc37be57324"} Jul 29 00:54:33.571367 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:33.571322 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-thq5l" event={"ID":"5719a514-b8de-4661-ab2a-fe0587726ae8","Type":"ContainerStarted","Data":"49a9052016c6dbef52d70ba6be032f195059388f72f1e9c8f267e608e878b3e3"} Jul 29 00:54:33.572992 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:33.572968 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" event={"ID":"e79868cd-1ac6-4262-ab1f-687d466f8b4d","Type":"ContainerStarted","Data":"8348ef7bd9e37672a35716945f54476af67427f8bf80fc9cc09f5f7d9eeef464"} Jul 29 00:54:33.574510 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:33.574484 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lsz6j" event={"ID":"47a76b9a-5cc4-4739-8ab3-2745dab6d404","Type":"ContainerStarted","Data":"ee54ac69dd1dd978a1ced7bf8d544f678d3300dd7d0691fd86a165be46b6ba4b"} Jul 29 00:54:33.577703 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:33.577682 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" event={"ID":"8a557028-eba5-431a-8690-dd95f88e82df","Type":"ContainerStarted","Data":"0bbea079001ad03555562a458a2241a2696651c9bb87731c4fffb22c94f1bd60"} Jul 29 00:54:33.577809 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:33.577748 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" event={"ID":"8a557028-eba5-431a-8690-dd95f88e82df","Type":"ContainerStarted","Data":"8a08668deb11bdf936be02ead1b36f9ed70bcf4d7bd11f7f321f7bd8ca541d23"} Jul 29 00:54:33.577809 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:33.577787 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" event={"ID":"8a557028-eba5-431a-8690-dd95f88e82df","Type":"ContainerStarted","Data":"0710a2d123d7d7297ca605434a35063fcaa0f51f00ff64bc085d2cf6100c3390"} Jul 29 00:54:33.579235 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:33.579202 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nqv9s" event={"ID":"a2127c7c-06fc-4849-ab98-2975c0862540","Type":"ContainerStarted","Data":"927c1ca440ed1e0d832e3ba41e6bd9d1fe753de7840f03d706ea562ce321faad"} Jul 29 00:54:33.580134 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:33.580091 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-136.ec2.internal" podStartSLOduration=22.580074871 podStartE2EDuration="22.580074871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:54:32.611409279 +0000 UTC m=+22.882053759" watchObservedRunningTime="2026-07-29 00:54:33.580074871 +0000 UTC m=+23.850719180" Jul 29 00:54:33.603274 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:33.603233 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j7xsc" podStartSLOduration=11.613699598 podStartE2EDuration="23.60322068s" totalImagesPullingTime="11.989521082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:54:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:54:33.57972918 +0000 UTC m=+23.850373490" watchObservedRunningTime="2026-07-29 00:54:33.60322068 +0000 UTC m=+23.873864987" Jul 29 00:54:33.616713 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:33.616675 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lsz6j" podStartSLOduration=2.346451083 podStartE2EDuration="22.616661057s" totalImagesPullingTime="20.270209974s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:54:33.616357646 +0000 UTC m=+23.887001952" watchObservedRunningTime="2026-07-29 00:54:33.616661057 +0000 UTC m=+23.887305788" Jul 29 00:54:33.642405 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:33.642372 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-thq5l" podStartSLOduration=3.698813597 podStartE2EDuration="23.642359744s" totalImagesPullingTime="19.943546147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:54:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:54:33.64202594 +0000 UTC m=+23.912670248" watchObservedRunningTime="2026-07-29 00:54:33.642359744 +0000 UTC m=+23.913004051" Jul 29 00:54:33.899851 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:33.899827 2543 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 29 00:54:34.275111 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:34.275004 2543 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T00:54:33.899846825Z","UUID":"44c26ea1-5496-4bb0-bbdf-91f2e176cafe","Handler":null,"Name":"","Endpoint":""} Jul 29 00:54:34.276593 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:34.276571 2543 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 29 00:54:34.276700 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:34.276611 2543 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 29 00:54:34.337892 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:34.337853 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:34.338064 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:34.337895 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:34.338064 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:34.337976 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:34.338160 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:34.338137 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lf9gq" podUID="03f35a91-81af-4c82-976b-50ad3b9d4ea6" Jul 29 00:54:34.338224 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:34.338198 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g87tb" podUID="926d37fa-f392-43d9-b446-160238607eff" Jul 29 00:54:34.338304 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:34.338282 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jx7tt" podUID="78bcb31a-2890-4527-b8f0-73cabeaecd3e" Jul 29 00:54:34.584279 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:34.584247 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" event={"ID":"e79868cd-1ac6-4262-ab1f-687d466f8b4d","Type":"ContainerStarted","Data":"0619753d53e559b58687d59b3c92b72a55c522df80cc7716a20463d399b2767f"} Jul 29 00:54:34.586130 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:34.586076 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-136.ec2.internal" event={"ID":"c032dfa2331a0d7e82f450dc05e5aa6c","Type":"ContainerStarted","Data":"dd0c6c9be685097c9ea843e8f0cc7f82f0ee88445cc24134f454e3c8681fafe6"} Jul 29 00:54:34.600529 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:34.600492 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nqv9s" podStartSLOduration=4.343636863 podStartE2EDuration="24.600478292s" totalImagesPullingTime="20.256841429s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:54:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:54:33.655014414 +0000 UTC m=+23.925658722" watchObservedRunningTime="2026-07-29 00:54:34.600478292 +0000 UTC m=+24.871122622" Jul 29 00:54:34.600648 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:34.600576 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-136.ec2.internal" podStartSLOduration=23.600571279 podStartE2EDuration="23.600571279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:54:34.600369221 +0000 UTC m=+24.871013528" watchObservedRunningTime="2026-07-29 00:54:34.600571279 +0000 UTC m=+24.871215587" Jul 29 00:54:34.868741 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:34.868660 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j7xsc" Jul 29 00:54:35.592192 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:35.592153 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" event={"ID":"8a557028-eba5-431a-8690-dd95f88e82df","Type":"ContainerStarted","Data":"339e349b62f21ef13cfcc4701906fec79f697a5719d53f0091b9b5f5833276a8"} Jul 29 00:54:35.594467 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:35.594431 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" event={"ID":"e79868cd-1ac6-4262-ab1f-687d466f8b4d","Type":"ContainerStarted","Data":"ba5bf76eb279cf874a5a84238c1262b7023db489666dbdd7837089c3ef2c70e0"} Jul 29 00:54:35.610619 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:35.610575 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68v7z" podStartSLOduration=2.889237334 podStartE2EDuration="25.610561656s" totalImagesPullingTime="22.721324322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:54:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:54:35.609152947 +0000 UTC m=+25.879797256" watchObservedRunningTime="2026-07-29 00:54:35.610561656 +0000 UTC m=+25.881205960" Jul 29 00:54:36.337247 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:36.337213 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:36.337523 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:36.337335 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:36.337523 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:36.337370 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:36.337523 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:36.337335 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g87tb" podUID="926d37fa-f392-43d9-b446-160238607eff" Jul 29 00:54:36.337523 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:36.337466 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jx7tt" podUID="78bcb31a-2890-4527-b8f0-73cabeaecd3e" Jul 29 00:54:36.337711 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:36.337561 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lf9gq" podUID="03f35a91-81af-4c82-976b-50ad3b9d4ea6" Jul 29 00:54:38.120207 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:38.120029 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j7xsc" Jul 29 00:54:38.120628 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:38.120608 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j7xsc" Jul 29 00:54:38.337635 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:38.337614 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:38.337737 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:38.337614 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:38.337789 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:38.337736 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lf9gq" podUID="03f35a91-81af-4c82-976b-50ad3b9d4ea6" Jul 29 00:54:38.337853 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:38.337807 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g87tb" podUID="926d37fa-f392-43d9-b446-160238607eff" Jul 29 00:54:38.337918 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:38.337876 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:38.338007 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:38.337985 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jx7tt" podUID="78bcb31a-2890-4527-b8f0-73cabeaecd3e" Jul 29 00:54:38.605938 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:38.605886 2543 generic.go:356] "Generic (PLEG): container finished" podID="1f974fab-a19b-419d-bc9f-ff551c5d928f" containerID="f9cd7c8c05713e8d217c770dbd925f4356fdac822f9792a11c1f2a2a1de1e8f7" exitCode=0 Jul 29 00:54:38.606104 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:38.605954 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7g65" event={"ID":"1f974fab-a19b-419d-bc9f-ff551c5d928f","Type":"ContainerDied","Data":"f9cd7c8c05713e8d217c770dbd925f4356fdac822f9792a11c1f2a2a1de1e8f7"} Jul 29 00:54:38.609516 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:38.609489 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" event={"ID":"8a557028-eba5-431a-8690-dd95f88e82df","Type":"ContainerStarted","Data":"a75369c3c4c4e7bc60eebc37b437f5722d3f927bbc70134abab049a8e930afdf"} Jul 29 00:54:38.610219 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:38.610203 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j7xsc" Jul 29 00:54:38.655565 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:38.655522 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" podStartSLOduration=7.905090311 podStartE2EDuration="28.655510341s" totalImagesPullingTime="20.75042003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:54:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:54:38.655109781 +0000 UTC m=+28.925754087" watchObservedRunningTime="2026-07-29 00:54:38.655510341 +0000 UTC m=+28.926154644" Jul 29 00:54:39.612454 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:39.612420 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:39.613206 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:39.612516 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:39.613206 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:39.612562 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:39.626444 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:39.626420 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:39.626676 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:39.626660 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:54:40.087009 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:40.086975 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lf9gq"] Jul 29 00:54:40.087169 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:40.087132 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:40.087261 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:40.087241 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lf9gq" podUID="03f35a91-81af-4c82-976b-50ad3b9d4ea6" Jul 29 00:54:40.089799 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:40.089772 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g87tb"] Jul 29 00:54:40.089931 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:40.089888 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:40.089992 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:40.089969 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g87tb" podUID="926d37fa-f392-43d9-b446-160238607eff" Jul 29 00:54:40.090455 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:40.090438 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jx7tt"] Jul 29 00:54:40.090536 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:40.090523 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:40.090653 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:40.090631 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jx7tt" podUID="78bcb31a-2890-4527-b8f0-73cabeaecd3e" Jul 29 00:54:40.615757 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:40.615725 2543 generic.go:356] "Generic (PLEG): container finished" podID="1f974fab-a19b-419d-bc9f-ff551c5d928f" containerID="d3e1e9cd84bb062ec87795daf89e27cb9e8845f0545357aab24c935a6f1a88aa" exitCode=0 Jul 29 00:54:40.616167 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:40.615811 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7g65" event={"ID":"1f974fab-a19b-419d-bc9f-ff551c5d928f","Type":"ContainerDied","Data":"d3e1e9cd84bb062ec87795daf89e27cb9e8845f0545357aab24c935a6f1a88aa"} Jul 29 00:54:41.337478 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:41.337452 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:41.337603 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:41.337558 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jx7tt" podUID="78bcb31a-2890-4527-b8f0-73cabeaecd3e" Jul 29 00:54:42.340835 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:42.340792 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:42.341204 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:42.340904 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lf9gq" podUID="03f35a91-81af-4c82-976b-50ad3b9d4ea6" Jul 29 00:54:42.341204 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:42.340963 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:42.341204 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:42.341047 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g87tb" podUID="926d37fa-f392-43d9-b446-160238607eff" Jul 29 00:54:42.621347 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:42.621319 2543 generic.go:356] "Generic (PLEG): container finished" podID="1f974fab-a19b-419d-bc9f-ff551c5d928f" containerID="da93672fafb8d805b081d15a93a8f4791c3069dd7135d1958103d14c390bb928" exitCode=0 Jul 29 00:54:42.621484 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:42.621374 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7g65" event={"ID":"1f974fab-a19b-419d-bc9f-ff551c5d928f","Type":"ContainerDied","Data":"da93672fafb8d805b081d15a93a8f4791c3069dd7135d1958103d14c390bb928"} Jul 29 00:54:42.917184 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:42.917153 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret\") pod \"global-pull-secret-syncer-lf9gq\" (UID: \"03f35a91-81af-4c82-976b-50ad3b9d4ea6\") " pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:42.917184 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:42.917188 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jspd\" (UniqueName: \"kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd\") pod \"network-check-target-g87tb\" (UID: \"926d37fa-f392-43d9-b446-160238607eff\") " pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:42.917343 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:42.917212 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs\") pod \"network-metrics-daemon-jx7tt\" (UID: \"78bcb31a-2890-4527-b8f0-73cabeaecd3e\") " pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:42.917343 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:42.917291 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 00:54:42.917343 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:42.917315 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 00:54:42.917343 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:42.917318 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 00:54:42.917343 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:42.917336 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 00:54:42.917496 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:42.917348 2543 projected.go:196] Error preparing data for projected volume kube-api-access-5jspd for pod openshift-network-diagnostics/network-check-target-g87tb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 00:54:42.917496 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:42.917353 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret podName:03f35a91-81af-4c82-976b-50ad3b9d4ea6 nodeName:}" failed. No retries permitted until 2026-07-29 00:55:14.91733886 +0000 UTC m=+65.187983149 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret") pod "global-pull-secret-syncer-lf9gq" (UID: "03f35a91-81af-4c82-976b-50ad3b9d4ea6") : object "kube-system"/"original-pull-secret" not registered Jul 29 00:54:42.917496 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:42.917370 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs podName:78bcb31a-2890-4527-b8f0-73cabeaecd3e nodeName:}" failed. No retries permitted until 2026-07-29 00:55:14.917361846 +0000 UTC m=+65.188006131 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs") pod "network-metrics-daemon-jx7tt" (UID: "78bcb31a-2890-4527-b8f0-73cabeaecd3e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 00:54:42.917496 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:42.917384 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd podName:926d37fa-f392-43d9-b446-160238607eff nodeName:}" failed. No retries permitted until 2026-07-29 00:55:14.917375839 +0000 UTC m=+65.188020145 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5jspd" (UniqueName: "kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd") pod "network-check-target-g87tb" (UID: "926d37fa-f392-43d9-b446-160238607eff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 00:54:43.336978 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:43.336953 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:43.337121 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:43.337064 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jx7tt" podUID="78bcb31a-2890-4527-b8f0-73cabeaecd3e" Jul 29 00:54:44.340953 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:44.340921 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:44.341339 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:44.340929 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:44.341339 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:44.341027 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g87tb" podUID="926d37fa-f392-43d9-b446-160238607eff" Jul 29 00:54:44.341339 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:44.341168 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lf9gq" podUID="03f35a91-81af-4c82-976b-50ad3b9d4ea6" Jul 29 00:54:45.037603 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.037518 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-136.ec2.internal" event="NodeReady" Jul 29 00:54:45.037766 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.037682 2543 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 29 00:54:45.069047 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.069006 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4"] Jul 29 00:54:45.070969 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.070946 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-684c9f4678-nd9lv"] Jul 29 00:54:45.071106 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.071076 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" Jul 29 00:54:45.073404 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.073375 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 29 00:54:45.073702 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.073684 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-bwc6w\"" Jul 29 00:54:45.074839 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.073947 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 29 00:54:45.075501 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.075476 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.077856 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.077837 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6dlx2\"" Jul 29 00:54:45.078088 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.078068 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 29 00:54:45.078195 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.078067 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 29 00:54:45.079430 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.079411 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 29 00:54:45.081567 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.081545 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4"] Jul 29 00:54:45.085104 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.085084 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 29 00:54:45.085416 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.085394 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-684c9f4678-nd9lv"] Jul 29 00:54:45.088707 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.088345 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h6ncm"] Jul 29 00:54:45.090001 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.089981 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7jhhc"] Jul 29 00:54:45.090094 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.090078 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h6ncm" Jul 29 00:54:45.093028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.092172 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7jhhc" Jul 29 00:54:45.093028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.092704 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-58969\"" Jul 29 00:54:45.093028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.092712 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 29 00:54:45.093028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.092779 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 29 00:54:45.093028 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.092866 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 29 00:54:45.094004 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.093986 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 29 00:54:45.094122 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.094009 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 29 00:54:45.094184 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.094126 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9mcp8\"" Jul 29 00:54:45.098726 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.098706 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h6ncm"] Jul 29 00:54:45.103108 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.102286 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7jhhc"] Jul 29 00:54:45.217125 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.216901 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j7xsc_0b304c44-8f9d-4830-a2e7-d5559eeb879f/konnectivity-agent/0.log" limit=1024 Jul 29 00:54:45.232644 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.232607 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3613abf5-ad33-4253-bf63-c8f7ccd99759-trusted-ca\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.232791 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.232645 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ab87c7b-6a40-4383-a3cb-90f40e463040-cert\") pod \"ingress-canary-h6ncm\" (UID: \"9ab87c7b-6a40-4383-a3cb-90f40e463040\") " pod="openshift-ingress-canary/ingress-canary-h6ncm" Jul 29 00:54:45.232791 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.232690 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9hls\" (UniqueName: \"kubernetes.io/projected/078d8065-0985-4266-b9ec-e607f7372bb7-kube-api-access-n9hls\") pod \"dns-default-7jhhc\" (UID: \"078d8065-0985-4266-b9ec-e607f7372bb7\") " pod="openshift-dns/dns-default-7jhhc" Jul 29 00:54:45.232791 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.232734 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-certificates\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.232791 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.232768 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3613abf5-ad33-4253-bf63-c8f7ccd99759-installation-pull-secrets\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.232992 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.232831 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-bound-sa-token\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.232992 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.232872 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls\") pod \"dns-default-7jhhc\" (UID: \"078d8065-0985-4266-b9ec-e607f7372bb7\") " pod="openshift-dns/dns-default-7jhhc" Jul 29 00:54:45.232992 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.232920 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078d8065-0985-4266-b9ec-e607f7372bb7-tmp-dir\") pod \"dns-default-7jhhc\" (UID: \"078d8065-0985-4266-b9ec-e607f7372bb7\") " pod="openshift-dns/dns-default-7jhhc" Jul 29 00:54:45.232992 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.232975 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.232992 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.232991 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b8a3faba-c784-4189-9c88-b6743126556a-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-b7qp4\" (UID: \"b8a3faba-c784-4189-9c88-b6743126556a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" Jul 29 00:54:45.233176 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.233034 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/078d8065-0985-4266-b9ec-e607f7372bb7-config-volume\") pod \"dns-default-7jhhc\" (UID: \"078d8065-0985-4266-b9ec-e607f7372bb7\") " pod="openshift-dns/dns-default-7jhhc" Jul 29 00:54:45.233176 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.233071 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3613abf5-ad33-4253-bf63-c8f7ccd99759-image-registry-private-configuration\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.233176 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.233090 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqm2x\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-kube-api-access-rqm2x\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.233176 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.233114 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-b7qp4\" (UID: \"b8a3faba-c784-4189-9c88-b6743126556a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" Jul 29 00:54:45.233342 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.233169 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ln2m\" (UniqueName: \"kubernetes.io/projected/9ab87c7b-6a40-4383-a3cb-90f40e463040-kube-api-access-7ln2m\") pod \"ingress-canary-h6ncm\" (UID: \"9ab87c7b-6a40-4383-a3cb-90f40e463040\") " pod="openshift-ingress-canary/ingress-canary-h6ncm" Jul 29 00:54:45.233342 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.233219 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3613abf5-ad33-4253-bf63-c8f7ccd99759-ca-trust-extracted\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.297447 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.297349 2543 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j7xsc_0b304c44-8f9d-4830-a2e7-d5559eeb879f/konnectivity-agent/0.log" limit=1024 Jul 29 00:54:45.334394 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.334360 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.334394 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.334399 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b8a3faba-c784-4189-9c88-b6743126556a-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-b7qp4\" (UID: \"b8a3faba-c784-4189-9c88-b6743126556a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" Jul 29 00:54:45.334621 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.334440 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/078d8065-0985-4266-b9ec-e607f7372bb7-config-volume\") pod \"dns-default-7jhhc\" (UID: \"078d8065-0985-4266-b9ec-e607f7372bb7\") " pod="openshift-dns/dns-default-7jhhc" Jul 29 00:54:45.334621 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.334483 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3613abf5-ad33-4253-bf63-c8f7ccd99759-image-registry-private-configuration\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.334621 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.334506 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqm2x\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-kube-api-access-rqm2x\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.334621 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:45.334518 2543 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 00:54:45.334621 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:45.334533 2543 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-684c9f4678-nd9lv: secret "image-registry-tls" not found Jul 29 00:54:45.334621 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.334532 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-b7qp4\" (UID: \"b8a3faba-c784-4189-9c88-b6743126556a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" Jul 29 00:54:45.334621 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:45.334599 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls podName:3613abf5-ad33-4253-bf63-c8f7ccd99759 nodeName:}" failed. No retries permitted until 2026-07-29 00:54:45.834580914 +0000 UTC m=+36.105225210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls") pod "image-registry-684c9f4678-nd9lv" (UID: "3613abf5-ad33-4253-bf63-c8f7ccd99759") : secret "image-registry-tls" not found Jul 29 00:54:45.334621 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.334604 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ln2m\" (UniqueName: \"kubernetes.io/projected/9ab87c7b-6a40-4383-a3cb-90f40e463040-kube-api-access-7ln2m\") pod \"ingress-canary-h6ncm\" (UID: \"9ab87c7b-6a40-4383-a3cb-90f40e463040\") " pod="openshift-ingress-canary/ingress-canary-h6ncm" Jul 29 00:54:45.335034 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.334632 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3613abf5-ad33-4253-bf63-c8f7ccd99759-ca-trust-extracted\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.335034 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.334681 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3613abf5-ad33-4253-bf63-c8f7ccd99759-trusted-ca\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.335034 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.334708 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ab87c7b-6a40-4383-a3cb-90f40e463040-cert\") pod \"ingress-canary-h6ncm\" (UID: \"9ab87c7b-6a40-4383-a3cb-90f40e463040\") " pod="openshift-ingress-canary/ingress-canary-h6ncm" Jul 29 00:54:45.335034 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:45.334636 2543 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 00:54:45.335034 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.334744 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9hls\" (UniqueName: \"kubernetes.io/projected/078d8065-0985-4266-b9ec-e607f7372bb7-kube-api-access-n9hls\") pod \"dns-default-7jhhc\" (UID: \"078d8065-0985-4266-b9ec-e607f7372bb7\") " pod="openshift-dns/dns-default-7jhhc" Jul 29 00:54:45.335034 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:45.334767 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert podName:b8a3faba-c784-4189-9c88-b6743126556a nodeName:}" failed. No retries permitted until 2026-07-29 00:54:45.834755488 +0000 UTC m=+36.105399792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-b7qp4" (UID: "b8a3faba-c784-4189-9c88-b6743126556a") : secret "networking-console-plugin-cert" not found Jul 29 00:54:45.335034 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.334785 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-certificates\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.335034 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.334827 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3613abf5-ad33-4253-bf63-c8f7ccd99759-installation-pull-secrets\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.335034 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.334884 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-bound-sa-token\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.335034 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.334931 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls\") pod \"dns-default-7jhhc\" (UID: \"078d8065-0985-4266-b9ec-e607f7372bb7\") " pod="openshift-dns/dns-default-7jhhc" Jul 29 00:54:45.335034 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.334984 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078d8065-0985-4266-b9ec-e607f7372bb7-tmp-dir\") pod \"dns-default-7jhhc\" (UID: \"078d8065-0985-4266-b9ec-e607f7372bb7\") " pod="openshift-dns/dns-default-7jhhc" Jul 29 00:54:45.335560 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.335056 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3613abf5-ad33-4253-bf63-c8f7ccd99759-ca-trust-extracted\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.335560 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.335136 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/078d8065-0985-4266-b9ec-e607f7372bb7-config-volume\") pod \"dns-default-7jhhc\" (UID: \"078d8065-0985-4266-b9ec-e607f7372bb7\") " pod="openshift-dns/dns-default-7jhhc" Jul 29 00:54:45.335560 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.335241 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b8a3faba-c784-4189-9c88-b6743126556a-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-b7qp4\" (UID: \"b8a3faba-c784-4189-9c88-b6743126556a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" Jul 29 00:54:45.335560 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.335261 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/078d8065-0985-4266-b9ec-e607f7372bb7-tmp-dir\") pod \"dns-default-7jhhc\" (UID: \"078d8065-0985-4266-b9ec-e607f7372bb7\") " pod="openshift-dns/dns-default-7jhhc" Jul 29 00:54:45.335560 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:45.335328 2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 00:54:45.335560 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:45.335383 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls podName:078d8065-0985-4266-b9ec-e607f7372bb7 nodeName:}" failed. No retries permitted until 2026-07-29 00:54:45.835366257 +0000 UTC m=+36.106010549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls") pod "dns-default-7jhhc" (UID: "078d8065-0985-4266-b9ec-e607f7372bb7") : secret "dns-default-metrics-tls" not found Jul 29 00:54:45.335788 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.335670 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3613abf5-ad33-4253-bf63-c8f7ccd99759-trusted-ca\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.335788 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.335737 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-certificates\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.336978 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.336957 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:54:45.339102 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.338921 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 00:54:45.339102 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.338976 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bxdbf\"" Jul 29 00:54:45.339854 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.339801 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3613abf5-ad33-4253-bf63-c8f7ccd99759-installation-pull-secrets\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.339952 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.339831 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ab87c7b-6a40-4383-a3cb-90f40e463040-cert\") pod \"ingress-canary-h6ncm\" (UID: \"9ab87c7b-6a40-4383-a3cb-90f40e463040\") " pod="openshift-ingress-canary/ingress-canary-h6ncm" Jul 29 00:54:45.339952 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.339836 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3613abf5-ad33-4253-bf63-c8f7ccd99759-image-registry-private-configuration\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.343254 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.343208 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-bound-sa-token\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.343591 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.343336 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ln2m\" (UniqueName: \"kubernetes.io/projected/9ab87c7b-6a40-4383-a3cb-90f40e463040-kube-api-access-7ln2m\") pod \"ingress-canary-h6ncm\" (UID: \"9ab87c7b-6a40-4383-a3cb-90f40e463040\") " pod="openshift-ingress-canary/ingress-canary-h6ncm" Jul 29 00:54:45.343591 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.343573 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9hls\" (UniqueName: \"kubernetes.io/projected/078d8065-0985-4266-b9ec-e607f7372bb7-kube-api-access-n9hls\") pod \"dns-default-7jhhc\" (UID: \"078d8065-0985-4266-b9ec-e607f7372bb7\") " pod="openshift-dns/dns-default-7jhhc" Jul 29 00:54:45.344009 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.343991 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqm2x\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-kube-api-access-rqm2x\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.404728 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.404700 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h6ncm" Jul 29 00:54:45.838834 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.838785 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:45.839019 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.838860 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-b7qp4\" (UID: \"b8a3faba-c784-4189-9c88-b6743126556a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" Jul 29 00:54:45.839019 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:45.838936 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls\") pod \"dns-default-7jhhc\" (UID: \"078d8065-0985-4266-b9ec-e607f7372bb7\") " pod="openshift-dns/dns-default-7jhhc" Jul 29 00:54:45.839019 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:45.838961 2543 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 00:54:45.839019 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:45.838980 2543 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-684c9f4678-nd9lv: secret "image-registry-tls" not found Jul 29 00:54:45.839183 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:45.839024 2543 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 00:54:45.839183 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:45.839035 2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 00:54:45.839183 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:45.839030 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls podName:3613abf5-ad33-4253-bf63-c8f7ccd99759 nodeName:}" failed. No retries permitted until 2026-07-29 00:54:46.839014871 +0000 UTC m=+37.109659170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls") pod "image-registry-684c9f4678-nd9lv" (UID: "3613abf5-ad33-4253-bf63-c8f7ccd99759") : secret "image-registry-tls" not found Jul 29 00:54:45.839183 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:45.839098 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls podName:078d8065-0985-4266-b9ec-e607f7372bb7 nodeName:}" failed. No retries permitted until 2026-07-29 00:54:46.839081226 +0000 UTC m=+37.109725516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls") pod "dns-default-7jhhc" (UID: "078d8065-0985-4266-b9ec-e607f7372bb7") : secret "dns-default-metrics-tls" not found Jul 29 00:54:45.839183 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:45.839115 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert podName:b8a3faba-c784-4189-9c88-b6743126556a nodeName:}" failed. No retries permitted until 2026-07-29 00:54:46.839103454 +0000 UTC m=+37.109747745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-b7qp4" (UID: "b8a3faba-c784-4189-9c88-b6743126556a") : secret "networking-console-plugin-cert" not found Jul 29 00:54:46.337797 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:46.337759 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:54:46.337797 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:46.337778 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:54:46.340077 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:46.340055 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 00:54:46.340799 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:46.340776 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ndg5h\"" Jul 29 00:54:46.340907 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:46.340800 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 00:54:46.340957 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:46.340776 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 00:54:46.747512 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:46.747486 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h6ncm"] Jul 29 00:54:46.750771 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:54:46.750738 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab87c7b_6a40_4383_a3cb_90f40e463040.slice/crio-3c93ad1ad3ec89a7080d11cc572a2b848e6beaa7baf108aff758f534b91a98fd WatchSource:0}: Error finding container 3c93ad1ad3ec89a7080d11cc572a2b848e6beaa7baf108aff758f534b91a98fd: Status 404 returned error can't find the container with id 3c93ad1ad3ec89a7080d11cc572a2b848e6beaa7baf108aff758f534b91a98fd Jul 29 00:54:46.852056 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:46.852031 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:46.852157 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:46.852104 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-b7qp4\" (UID: \"b8a3faba-c784-4189-9c88-b6743126556a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" Jul 29 00:54:46.852209 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:46.852180 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls\") pod \"dns-default-7jhhc\" (UID: \"078d8065-0985-4266-b9ec-e607f7372bb7\") " pod="openshift-dns/dns-default-7jhhc" Jul 29 00:54:46.852260 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:46.852184 2543 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 00:54:46.852260 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:46.852230 2543 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-684c9f4678-nd9lv: secret "image-registry-tls" not found Jul 29 00:54:46.852260 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:46.852250 2543 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 00:54:46.852382 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:46.852258 2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 00:54:46.852382 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:46.852298 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls podName:3613abf5-ad33-4253-bf63-c8f7ccd99759 nodeName:}" failed. No retries permitted until 2026-07-29 00:54:48.852279372 +0000 UTC m=+39.122923681 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls") pod "image-registry-684c9f4678-nd9lv" (UID: "3613abf5-ad33-4253-bf63-c8f7ccd99759") : secret "image-registry-tls" not found Jul 29 00:54:46.852382 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:46.852316 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert podName:b8a3faba-c784-4189-9c88-b6743126556a nodeName:}" failed. No retries permitted until 2026-07-29 00:54:48.852306966 +0000 UTC m=+39.122951258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-b7qp4" (UID: "b8a3faba-c784-4189-9c88-b6743126556a") : secret "networking-console-plugin-cert" not found Jul 29 00:54:46.852382 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:46.852330 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls podName:078d8065-0985-4266-b9ec-e607f7372bb7 nodeName:}" failed. No retries permitted until 2026-07-29 00:54:48.852322644 +0000 UTC m=+39.122966933 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls") pod "dns-default-7jhhc" (UID: "078d8065-0985-4266-b9ec-e607f7372bb7") : secret "dns-default-metrics-tls" not found Jul 29 00:54:47.636338 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:47.636120 2543 generic.go:356] "Generic (PLEG): container finished" podID="1f974fab-a19b-419d-bc9f-ff551c5d928f" containerID="c25a6600d5eb25e4fa05e4fcef6bd9ef6bc70648f18fc8a3e4a31f3708c5b33f" exitCode=0 Jul 29 00:54:47.636511 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:47.636199 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7g65" event={"ID":"1f974fab-a19b-419d-bc9f-ff551c5d928f","Type":"ContainerDied","Data":"c25a6600d5eb25e4fa05e4fcef6bd9ef6bc70648f18fc8a3e4a31f3708c5b33f"} Jul 29 00:54:47.637617 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:47.637590 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h6ncm" event={"ID":"9ab87c7b-6a40-4383-a3cb-90f40e463040","Type":"ContainerStarted","Data":"3c93ad1ad3ec89a7080d11cc572a2b848e6beaa7baf108aff758f534b91a98fd"} Jul 29 00:54:48.642045 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:48.642005 2543 generic.go:356] "Generic (PLEG): container finished" podID="1f974fab-a19b-419d-bc9f-ff551c5d928f" containerID="8e5e5b3fc806f286d7497c5c30553d919609bf0a9e7d42714196b37c241b6832" exitCode=0 Jul 29 00:54:48.642413 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:48.642072 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7g65" event={"ID":"1f974fab-a19b-419d-bc9f-ff551c5d928f","Type":"ContainerDied","Data":"8e5e5b3fc806f286d7497c5c30553d919609bf0a9e7d42714196b37c241b6832"} Jul 29 00:54:48.871446 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:48.871408 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls\") pod \"dns-default-7jhhc\" (UID: \"078d8065-0985-4266-b9ec-e607f7372bb7\") " pod="openshift-dns/dns-default-7jhhc" Jul 29 00:54:48.871639 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:48.871461 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:48.871639 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:48.871520 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-b7qp4\" (UID: \"b8a3faba-c784-4189-9c88-b6743126556a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" Jul 29 00:54:48.871639 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:48.871588 2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 00:54:48.871639 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:48.871625 2543 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 00:54:48.871839 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:48.871654 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls podName:078d8065-0985-4266-b9ec-e607f7372bb7 nodeName:}" failed. No retries permitted until 2026-07-29 00:54:52.87163432 +0000 UTC m=+43.142278628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls") pod "dns-default-7jhhc" (UID: "078d8065-0985-4266-b9ec-e607f7372bb7") : secret "dns-default-metrics-tls" not found Jul 29 00:54:48.871839 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:48.871671 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert podName:b8a3faba-c784-4189-9c88-b6743126556a nodeName:}" failed. No retries permitted until 2026-07-29 00:54:52.871662555 +0000 UTC m=+43.142306844 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-b7qp4" (UID: "b8a3faba-c784-4189-9c88-b6743126556a") : secret "networking-console-plugin-cert" not found Jul 29 00:54:48.871839 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:48.871681 2543 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 00:54:48.871839 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:48.871704 2543 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-684c9f4678-nd9lv: secret "image-registry-tls" not found Jul 29 00:54:48.871839 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:48.871760 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls podName:3613abf5-ad33-4253-bf63-c8f7ccd99759 nodeName:}" failed. No retries permitted until 2026-07-29 00:54:52.871741678 +0000 UTC m=+43.142385986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls") pod "image-registry-684c9f4678-nd9lv" (UID: "3613abf5-ad33-4253-bf63-c8f7ccd99759") : secret "image-registry-tls" not found Jul 29 00:54:49.647307 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:49.647280 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7g65" event={"ID":"1f974fab-a19b-419d-bc9f-ff551c5d928f","Type":"ContainerStarted","Data":"3fce7786beb3ae7c7a9c35023ad914988738adab308534d45580b42fdb6750cd"} Jul 29 00:54:49.669135 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:49.669083 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d7g65" podStartSLOduration=8.436238298 podStartE2EDuration="39.669063988s" totalImagesPullingTime="31.23282569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:54:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:54:49.668172384 +0000 UTC m=+39.938816695" watchObservedRunningTime="2026-07-29 00:54:49.669063988 +0000 UTC m=+39.939708292" Jul 29 00:54:50.650441 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:50.650403 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h6ncm" event={"ID":"9ab87c7b-6a40-4383-a3cb-90f40e463040","Type":"ContainerStarted","Data":"4d2b5b7bee814f032266069a18414eb3253fb141dbb1d5fbdd296c4acabc78ac"} Jul 29 00:54:50.664103 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:50.664055 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h6ncm" podStartSLOduration=2.8434751560000002 podStartE2EDuration="5.6640406s" totalImagesPullingTime="2.820565444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:54:50.663320625 +0000 UTC m=+40.933964931" watchObservedRunningTime="2026-07-29 00:54:50.6640406 +0000 UTC m=+40.934684907" Jul 29 00:54:52.902166 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:52.902129 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls\") pod \"dns-default-7jhhc\" (UID: \"078d8065-0985-4266-b9ec-e607f7372bb7\") " pod="openshift-dns/dns-default-7jhhc" Jul 29 00:54:52.902526 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:52.902178 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:54:52.902526 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:54:52.902220 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-b7qp4\" (UID: \"b8a3faba-c784-4189-9c88-b6743126556a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" Jul 29 00:54:52.902526 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:52.902285 2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 00:54:52.902526 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:52.902328 2543 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 00:54:52.902526 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:52.902336 2543 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 00:54:52.902526 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:52.902352 2543 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-684c9f4678-nd9lv: secret "image-registry-tls" not found Jul 29 00:54:52.902526 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:52.902355 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls podName:078d8065-0985-4266-b9ec-e607f7372bb7 nodeName:}" failed. No retries permitted until 2026-07-29 00:55:00.902332921 +0000 UTC m=+51.172977219 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls") pod "dns-default-7jhhc" (UID: "078d8065-0985-4266-b9ec-e607f7372bb7") : secret "dns-default-metrics-tls" not found Jul 29 00:54:52.902526 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:52.902372 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert podName:b8a3faba-c784-4189-9c88-b6743126556a nodeName:}" failed. No retries permitted until 2026-07-29 00:55:00.902365474 +0000 UTC m=+51.173009760 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-b7qp4" (UID: "b8a3faba-c784-4189-9c88-b6743126556a") : secret "networking-console-plugin-cert" not found Jul 29 00:54:52.902526 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:54:52.902384 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls podName:3613abf5-ad33-4253-bf63-c8f7ccd99759 nodeName:}" failed. No retries permitted until 2026-07-29 00:55:00.902379166 +0000 UTC m=+51.173023451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls") pod "image-registry-684c9f4678-nd9lv" (UID: "3613abf5-ad33-4253-bf63-c8f7ccd99759") : secret "image-registry-tls" not found Jul 29 00:55:00.968187 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:00.968150 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls\") pod \"dns-default-7jhhc\" (UID: \"078d8065-0985-4266-b9ec-e607f7372bb7\") " pod="openshift-dns/dns-default-7jhhc" Jul 29 00:55:00.968683 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:00.968209 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:55:00.968683 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:00.968313 2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 00:55:00.968683 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:00.968366 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-b7qp4\" (UID: \"b8a3faba-c784-4189-9c88-b6743126556a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" Jul 29 00:55:00.968683 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:00.968388 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls podName:078d8065-0985-4266-b9ec-e607f7372bb7 nodeName:}" failed. No retries permitted until 2026-07-29 00:55:16.968371761 +0000 UTC m=+67.239016046 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls") pod "dns-default-7jhhc" (UID: "078d8065-0985-4266-b9ec-e607f7372bb7") : secret "dns-default-metrics-tls" not found Jul 29 00:55:00.968683 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:00.968317 2543 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 00:55:00.968683 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:00.968429 2543 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-684c9f4678-nd9lv: secret "image-registry-tls" not found Jul 29 00:55:00.968683 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:00.968432 2543 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 00:55:00.968683 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:00.968481 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls podName:3613abf5-ad33-4253-bf63-c8f7ccd99759 nodeName:}" failed. No retries permitted until 2026-07-29 00:55:16.968463942 +0000 UTC m=+67.239108241 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls") pod "image-registry-684c9f4678-nd9lv" (UID: "3613abf5-ad33-4253-bf63-c8f7ccd99759") : secret "image-registry-tls" not found Jul 29 00:55:00.968683 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:00.968503 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert podName:b8a3faba-c784-4189-9c88-b6743126556a nodeName:}" failed. No retries permitted until 2026-07-29 00:55:16.968491514 +0000 UTC m=+67.239135801 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-b7qp4" (UID: "b8a3faba-c784-4189-9c88-b6743126556a") : secret "networking-console-plugin-cert" not found Jul 29 00:55:11.632108 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:11.632072 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h8nv4" Jul 29 00:55:14.963488 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:14.963443 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret\") pod \"global-pull-secret-syncer-lf9gq\" (UID: \"03f35a91-81af-4c82-976b-50ad3b9d4ea6\") " pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:55:14.963488 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:14.963494 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jspd\" (UniqueName: \"kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd\") pod \"network-check-target-g87tb\" (UID: \"926d37fa-f392-43d9-b446-160238607eff\") " pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:55:14.964000 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:14.963669 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs\") pod \"network-metrics-daemon-jx7tt\" (UID: \"78bcb31a-2890-4527-b8f0-73cabeaecd3e\") " pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:55:14.967654 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:14.967637 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 00:55:14.967725 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:14.967709 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 00:55:14.969023 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:14.969011 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 00:55:14.974526 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:14.974509 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 29 00:55:14.974577 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:14.974568 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs podName:78bcb31a-2890-4527-b8f0-73cabeaecd3e nodeName:}" failed. No retries permitted until 2026-07-29 00:56:18.97455303 +0000 UTC m=+129.245197316 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs") pod "network-metrics-daemon-jx7tt" (UID: "78bcb31a-2890-4527-b8f0-73cabeaecd3e") : secret "metrics-daemon-secret" not found Jul 29 00:55:14.976224 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:14.976209 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 00:55:14.977650 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:14.977629 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/03f35a91-81af-4c82-976b-50ad3b9d4ea6-original-pull-secret\") pod \"global-pull-secret-syncer-lf9gq\" (UID: \"03f35a91-81af-4c82-976b-50ad3b9d4ea6\") " pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:55:14.986644 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:14.986620 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jspd\" (UniqueName: \"kubernetes.io/projected/926d37fa-f392-43d9-b446-160238607eff-kube-api-access-5jspd\") pod \"network-check-target-g87tb\" (UID: \"926d37fa-f392-43d9-b446-160238607eff\") " pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:55:15.149505 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:15.149470 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lf9gq" Jul 29 00:55:15.156682 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:15.156657 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ndg5h\"" Jul 29 00:55:15.165367 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:15.165336 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:55:15.272544 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:15.272520 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lf9gq"] Jul 29 00:55:15.275055 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:55:15.275024 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f35a91_81af_4c82_976b_50ad3b9d4ea6.slice/crio-242bc707587d7c82403a3d8380af42d81a4c312032e4ab0297164bb81157bc30 WatchSource:0}: Error finding container 242bc707587d7c82403a3d8380af42d81a4c312032e4ab0297164bb81157bc30: Status 404 returned error can't find the container with id 242bc707587d7c82403a3d8380af42d81a4c312032e4ab0297164bb81157bc30 Jul 29 00:55:15.285833 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:15.285791 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g87tb"] Jul 29 00:55:15.288654 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:55:15.288631 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926d37fa_f392_43d9_b446_160238607eff.slice/crio-fd488952a2b73a7ba90596ccc4b896f85dfafdf02db623ff0ca7cf7aaf3e88f6 WatchSource:0}: Error finding container fd488952a2b73a7ba90596ccc4b896f85dfafdf02db623ff0ca7cf7aaf3e88f6: Status 404 returned error can't find the container with id fd488952a2b73a7ba90596ccc4b896f85dfafdf02db623ff0ca7cf7aaf3e88f6 Jul 29 00:55:15.713138 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:15.713093 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lf9gq" event={"ID":"03f35a91-81af-4c82-976b-50ad3b9d4ea6","Type":"ContainerStarted","Data":"242bc707587d7c82403a3d8380af42d81a4c312032e4ab0297164bb81157bc30"} Jul 29 00:55:15.714229 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:15.714186 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g87tb" event={"ID":"926d37fa-f392-43d9-b446-160238607eff","Type":"ContainerStarted","Data":"fd488952a2b73a7ba90596ccc4b896f85dfafdf02db623ff0ca7cf7aaf3e88f6"} Jul 29 00:55:16.983034 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:16.982994 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:55:16.983505 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:16.983054 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-b7qp4\" (UID: \"b8a3faba-c784-4189-9c88-b6743126556a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" Jul 29 00:55:16.983505 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:16.983165 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls\") pod \"dns-default-7jhhc\" (UID: \"078d8065-0985-4266-b9ec-e607f7372bb7\") " pod="openshift-dns/dns-default-7jhhc" Jul 29 00:55:16.983505 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:16.983252 2543 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 00:55:16.983505 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:16.983277 2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 00:55:16.983505 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:16.983282 2543 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 00:55:16.983505 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:16.983302 2543 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-684c9f4678-nd9lv: secret "image-registry-tls" not found Jul 29 00:55:16.983505 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:16.983327 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert podName:b8a3faba-c784-4189-9c88-b6743126556a nodeName:}" failed. No retries permitted until 2026-07-29 00:55:48.983306795 +0000 UTC m=+99.253951085 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-b7qp4" (UID: "b8a3faba-c784-4189-9c88-b6743126556a") : secret "networking-console-plugin-cert" not found Jul 29 00:55:16.983505 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:16.983351 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls podName:3613abf5-ad33-4253-bf63-c8f7ccd99759 nodeName:}" failed. No retries permitted until 2026-07-29 00:55:48.98333423 +0000 UTC m=+99.253978534 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls") pod "image-registry-684c9f4678-nd9lv" (UID: "3613abf5-ad33-4253-bf63-c8f7ccd99759") : secret "image-registry-tls" not found Jul 29 00:55:16.983505 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:16.983368 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls podName:078d8065-0985-4266-b9ec-e607f7372bb7 nodeName:}" failed. No retries permitted until 2026-07-29 00:55:48.983358802 +0000 UTC m=+99.254003092 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls") pod "dns-default-7jhhc" (UID: "078d8065-0985-4266-b9ec-e607f7372bb7") : secret "dns-default-metrics-tls" not found Jul 29 00:55:20.728466 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:20.728424 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lf9gq" event={"ID":"03f35a91-81af-4c82-976b-50ad3b9d4ea6","Type":"ContainerStarted","Data":"d57e95fa52187213d88e91df3425899879caf20607cb4c69016a64950d784bfe"} Jul 29 00:55:20.729990 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:20.729960 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g87tb" event={"ID":"926d37fa-f392-43d9-b446-160238607eff","Type":"ContainerStarted","Data":"b3a8c14fc3093ab416c24051409d3d964c3d14c20ab110e18701c09e2cccbfb0"} Jul 29 00:55:20.730097 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:20.730079 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:55:20.741621 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:20.741580 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lf9gq" podStartSLOduration=64.924585897 podStartE2EDuration="1m9.74156779s" totalImagesPullingTime="4.816981893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:55:20.740962042 +0000 UTC m=+71.011606376" watchObservedRunningTime="2026-07-29 00:55:20.74156779 +0000 UTC m=+71.012212097" Jul 29 00:55:20.754068 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:20.754031 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g87tb" podStartSLOduration=65.962296895 podStartE2EDuration="1m10.754019283s" totalImagesPullingTime="4.791722388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:54:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:55:20.753121264 +0000 UTC m=+71.023765572" watchObservedRunningTime="2026-07-29 00:55:20.754019283 +0000 UTC m=+71.024663583" Jul 29 00:55:49.002012 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:49.001872 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls\") pod \"dns-default-7jhhc\" (UID: \"078d8065-0985-4266-b9ec-e607f7372bb7\") " pod="openshift-dns/dns-default-7jhhc" Jul 29 00:55:49.002012 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:49.001938 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:55:49.002012 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:49.001963 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-b7qp4\" (UID: \"b8a3faba-c784-4189-9c88-b6743126556a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" Jul 29 00:55:49.002592 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:49.002034 2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 00:55:49.002592 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:49.002085 2543 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 00:55:49.002592 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:49.002101 2543 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-684c9f4678-nd9lv: secret "image-registry-tls" not found Jul 29 00:55:49.002592 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:49.002114 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls podName:078d8065-0985-4266-b9ec-e607f7372bb7 nodeName:}" failed. No retries permitted until 2026-07-29 00:56:53.002092459 +0000 UTC m=+163.272736744 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls") pod "dns-default-7jhhc" (UID: "078d8065-0985-4266-b9ec-e607f7372bb7") : secret "dns-default-metrics-tls" not found Jul 29 00:55:49.002592 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:49.002040 2543 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 00:55:49.002592 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:49.002151 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls podName:3613abf5-ad33-4253-bf63-c8f7ccd99759 nodeName:}" failed. No retries permitted until 2026-07-29 00:56:53.002139898 +0000 UTC m=+163.272784198 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls") pod "image-registry-684c9f4678-nd9lv" (UID: "3613abf5-ad33-4253-bf63-c8f7ccd99759") : secret "image-registry-tls" not found Jul 29 00:55:49.002592 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:55:49.002178 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert podName:b8a3faba-c784-4189-9c88-b6743126556a nodeName:}" failed. No retries permitted until 2026-07-29 00:56:53.002165413 +0000 UTC m=+163.272809715 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-b7qp4" (UID: "b8a3faba-c784-4189-9c88-b6743126556a") : secret "networking-console-plugin-cert" not found Jul 29 00:55:51.734213 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:55:51.734184 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g87tb" Jul 29 00:56:19.012753 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:56:19.012697 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs\") pod \"network-metrics-daemon-jx7tt\" (UID: \"78bcb31a-2890-4527-b8f0-73cabeaecd3e\") " pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:56:19.013273 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:56:19.012861 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 29 00:56:19.013273 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:56:19.012939 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs podName:78bcb31a-2890-4527-b8f0-73cabeaecd3e nodeName:}" failed. No retries permitted until 2026-07-29 00:58:21.012920167 +0000 UTC m=+251.283564474 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs") pod "network-metrics-daemon-jx7tt" (UID: "78bcb31a-2890-4527-b8f0-73cabeaecd3e") : secret "metrics-daemon-secret" not found Jul 29 00:56:48.086609 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:56:48.086557 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" podUID="b8a3faba-c784-4189-9c88-b6743126556a" Jul 29 00:56:48.094758 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:56:48.094730 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" podUID="3613abf5-ad33-4253-bf63-c8f7ccd99759" Jul 29 00:56:48.110896 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:56:48.110866 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7jhhc" podUID="078d8065-0985-4266-b9ec-e607f7372bb7" Jul 29 00:56:48.359848 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:56:48.359728 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-jx7tt" podUID="78bcb31a-2890-4527-b8f0-73cabeaecd3e" Jul 29 00:56:48.939767 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:56:48.939744 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" Jul 29 00:56:48.939930 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:56:48.939747 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7jhhc" Jul 29 00:56:48.939930 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:56:48.939747 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:56:53.065949 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:56:53.065904 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls\") pod \"dns-default-7jhhc\" (UID: \"078d8065-0985-4266-b9ec-e607f7372bb7\") " pod="openshift-dns/dns-default-7jhhc" Jul 29 00:56:53.065949 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:56:53.065962 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:56:53.066406 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:56:53.065991 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-b7qp4\" (UID: \"b8a3faba-c784-4189-9c88-b6743126556a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" Jul 29 00:56:53.066406 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:56:53.066065 2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 00:56:53.066406 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:56:53.066089 2543 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 29 00:56:53.066406 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:56:53.066113 2543 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 29 00:56:53.066406 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:56:53.066128 2543 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-684c9f4678-nd9lv: secret "image-registry-tls" not found Jul 29 00:56:53.066406 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:56:53.066139 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls podName:078d8065-0985-4266-b9ec-e607f7372bb7 nodeName:}" failed. No retries permitted until 2026-07-29 00:58:55.066123691 +0000 UTC m=+285.336767976 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls") pod "dns-default-7jhhc" (UID: "078d8065-0985-4266-b9ec-e607f7372bb7") : secret "dns-default-metrics-tls" not found Jul 29 00:56:53.066406 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:56:53.066153 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert podName:b8a3faba-c784-4189-9c88-b6743126556a nodeName:}" failed. No retries permitted until 2026-07-29 00:58:55.066146601 +0000 UTC m=+285.336790885 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-b7qp4" (UID: "b8a3faba-c784-4189-9c88-b6743126556a") : secret "networking-console-plugin-cert" not found Jul 29 00:56:53.066406 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:56:53.066185 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls podName:3613abf5-ad33-4253-bf63-c8f7ccd99759 nodeName:}" failed. No retries permitted until 2026-07-29 00:58:55.066172942 +0000 UTC m=+285.336817228 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls") pod "image-registry-684c9f4678-nd9lv" (UID: "3613abf5-ad33-4253-bf63-c8f7ccd99759") : secret "image-registry-tls" not found Jul 29 00:57:03.338034 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:57:03.337928 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:58:21.091386 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:21.091344 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs\") pod \"network-metrics-daemon-jx7tt\" (UID: \"78bcb31a-2890-4527-b8f0-73cabeaecd3e\") " pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:58:21.091934 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:21.091471 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 29 00:58:21.091934 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:21.091522 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs podName:78bcb31a-2890-4527-b8f0-73cabeaecd3e nodeName:}" failed. No retries permitted until 2026-07-29 01:00:23.091508905 +0000 UTC m=+373.362153189 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs") pod "network-metrics-daemon-jx7tt" (UID: "78bcb31a-2890-4527-b8f0-73cabeaecd3e") : secret "metrics-daemon-secret" not found Jul 29 00:58:29.421468 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.421435 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-v9h56"] Jul 29 00:58:29.423879 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.423848 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-vvwkz"] Jul 29 00:58:29.425191 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.425170 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-v9h56" Jul 29 00:58:29.426868 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.426848 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq"] Jul 29 00:58:29.428094 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.428068 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-225bq\"" Jul 29 00:58:29.428939 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.428263 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vvwkz" Jul 29 00:58:29.428939 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.428493 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 29 00:58:29.428939 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.428648 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 29 00:58:29.428939 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.428761 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 29 00:58:29.428939 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.428910 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 29 00:58:29.429875 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.429853 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-dl64r"] Jul 29 00:58:29.429972 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.429954 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq" Jul 29 00:58:29.431809 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.431790 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" Jul 29 00:58:29.432079 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.431791 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vnhgj\"" Jul 29 00:58:29.432079 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.432065 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 29 00:58:29.432321 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.432305 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 29 00:58:29.432384 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.432345 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-59r9c\"" Jul 29 00:58:29.432552 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.432537 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 29 00:58:29.432620 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.432564 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 29 00:58:29.433604 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.433590 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 29 00:58:29.433938 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.433919 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 29 00:58:29.434175 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.434160 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 29 00:58:29.434219 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.434172 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 29 00:58:29.434644 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.434630 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5vv4s\"" Jul 29 00:58:29.439416 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.439316 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-vvwkz"] Jul 29 00:58:29.440449 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.440429 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq"] Jul 29 00:58:29.441517 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.441500 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-v9h56"] Jul 29 00:58:29.442425 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.442408 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-dl64r"] Jul 29 00:58:29.442985 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.442964 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 29 00:58:29.448302 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.448279 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6ed4d34-e21f-445c-850b-2ee2e32aa6f0-serving-cert\") pod \"console-operator-584dcb6b54-dl64r\" (UID: \"c6ed4d34-e21f-445c-850b-2ee2e32aa6f0\") " pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" Jul 29 00:58:29.448393 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.448321 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snjvk\" (UniqueName: \"kubernetes.io/projected/d3f4c767-973d-46ca-b0d9-baf948f97b4e-kube-api-access-snjvk\") pod \"network-check-source-7d88cb55d8-vvwkz\" (UID: \"d3f4c767-973d-46ca-b0d9-baf948f97b4e\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vvwkz" Jul 29 00:58:29.448393 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.448354 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20eff4d6-fdde-4bb9-a489-80daad0e2321-serving-cert\") pod \"service-ca-operator-5d6c75f84f-v9h56\" (UID: \"20eff4d6-fdde-4bb9-a489-80daad0e2321\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-v9h56" Jul 29 00:58:29.448393 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.448379 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6ed4d34-e21f-445c-850b-2ee2e32aa6f0-config\") pod \"console-operator-584dcb6b54-dl64r\" (UID: \"c6ed4d34-e21f-445c-850b-2ee2e32aa6f0\") " pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" Jul 29 00:58:29.448545 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.448414 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qt8z\" (UniqueName: \"kubernetes.io/projected/c6ed4d34-e21f-445c-850b-2ee2e32aa6f0-kube-api-access-6qt8z\") pod \"console-operator-584dcb6b54-dl64r\" (UID: \"c6ed4d34-e21f-445c-850b-2ee2e32aa6f0\") " pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" Jul 29 00:58:29.448545 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.448435 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6ed4d34-e21f-445c-850b-2ee2e32aa6f0-trusted-ca\") pod \"console-operator-584dcb6b54-dl64r\" (UID: \"c6ed4d34-e21f-445c-850b-2ee2e32aa6f0\") " pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" Jul 29 00:58:29.448545 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.448514 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20eff4d6-fdde-4bb9-a489-80daad0e2321-config\") pod \"service-ca-operator-5d6c75f84f-v9h56\" (UID: \"20eff4d6-fdde-4bb9-a489-80daad0e2321\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-v9h56" Jul 29 00:58:29.448686 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.448547 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/842fabe6-40df-4c93-bba3-568afdaa041e-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq\" (UID: \"842fabe6-40df-4c93-bba3-568afdaa041e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq" Jul 29 00:58:29.448686 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.448598 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpvl9\" (UniqueName: \"kubernetes.io/projected/842fabe6-40df-4c93-bba3-568afdaa041e-kube-api-access-cpvl9\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq\" (UID: \"842fabe6-40df-4c93-bba3-568afdaa041e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq" Jul 29 00:58:29.448686 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.448626 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq699\" (UniqueName: \"kubernetes.io/projected/20eff4d6-fdde-4bb9-a489-80daad0e2321-kube-api-access-fq699\") pod \"service-ca-operator-5d6c75f84f-v9h56\" (UID: \"20eff4d6-fdde-4bb9-a489-80daad0e2321\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-v9h56" Jul 29 00:58:29.448864 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.448692 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/842fabe6-40df-4c93-bba3-568afdaa041e-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq\" (UID: \"842fabe6-40df-4c93-bba3-568afdaa041e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq" Jul 29 00:58:29.517907 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.517873 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx"] Jul 29 00:58:29.520018 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.519993 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wb72f"] Jul 29 00:58:29.520138 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.520114 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx" Jul 29 00:58:29.522111 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.522091 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7d6d7bcf7c-62rxf"] Jul 29 00:58:29.522286 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.522268 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wb72f" Jul 29 00:58:29.522789 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.522770 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 29 00:58:29.522929 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.522771 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-nlwlq\"" Jul 29 00:58:29.522929 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.522917 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 29 00:58:29.523049 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.522992 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 29 00:58:29.523049 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.523026 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 29 00:58:29.523976 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.523962 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:29.524523 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.524500 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 29 00:58:29.524633 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.524523 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 29 00:58:29.524633 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.524532 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qzrjq\"" Jul 29 00:58:29.524633 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.524527 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 29 00:58:29.526186 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.526166 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 29 00:58:29.526274 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.526256 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 29 00:58:29.526274 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.526267 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 29 00:58:29.526392 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.526287 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 29 00:58:29.526778 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.526761 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 29 00:58:29.527531 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.527514 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zftl7\"" Jul 29 00:58:29.527982 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.527832 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 29 00:58:29.532241 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.532219 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wb72f"] Jul 29 00:58:29.533074 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.533054 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx"] Jul 29 00:58:29.540533 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.540511 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7d6d7bcf7c-62rxf"] Jul 29 00:58:29.549209 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.549187 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qt8z\" (UniqueName: \"kubernetes.io/projected/c6ed4d34-e21f-445c-850b-2ee2e32aa6f0-kube-api-access-6qt8z\") pod \"console-operator-584dcb6b54-dl64r\" (UID: \"c6ed4d34-e21f-445c-850b-2ee2e32aa6f0\") " pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" Jul 29 00:58:29.549318 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.549228 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-service-ca-bundle\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:29.549318 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.549249 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6ed4d34-e21f-445c-850b-2ee2e32aa6f0-trusted-ca\") pod \"console-operator-584dcb6b54-dl64r\" (UID: \"c6ed4d34-e21f-445c-850b-2ee2e32aa6f0\") " pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" Jul 29 00:58:29.549318 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.549269 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-default-certificate\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:29.549481 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.549324 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20eff4d6-fdde-4bb9-a489-80daad0e2321-config\") pod \"service-ca-operator-5d6c75f84f-v9h56\" (UID: \"20eff4d6-fdde-4bb9-a489-80daad0e2321\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-v9h56" Jul 29 00:58:29.549481 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.549364 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/842fabe6-40df-4c93-bba3-568afdaa041e-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq\" (UID: \"842fabe6-40df-4c93-bba3-568afdaa041e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq" Jul 29 00:58:29.549481 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.549445 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79skp\" (UniqueName: \"kubernetes.io/projected/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-kube-api-access-79skp\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:29.549627 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.549483 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-zt4lx\" (UID: \"90ebb0ff-2b60-4c52-97cb-1cf589c21b61\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx" Jul 29 00:58:29.549627 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.549540 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpvl9\" (UniqueName: \"kubernetes.io/projected/842fabe6-40df-4c93-bba3-568afdaa041e-kube-api-access-cpvl9\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq\" (UID: \"842fabe6-40df-4c93-bba3-568afdaa041e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq" Jul 29 00:58:29.549627 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.549576 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq699\" (UniqueName: \"kubernetes.io/projected/20eff4d6-fdde-4bb9-a489-80daad0e2321-kube-api-access-fq699\") pod \"service-ca-operator-5d6c75f84f-v9h56\" (UID: \"20eff4d6-fdde-4bb9-a489-80daad0e2321\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-v9h56" Jul 29 00:58:29.549627 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.549614 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/842fabe6-40df-4c93-bba3-568afdaa041e-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq\" (UID: \"842fabe6-40df-4c93-bba3-568afdaa041e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq" Jul 29 00:58:29.549861 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.549656 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tbbs\" (UniqueName: \"kubernetes.io/projected/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-kube-api-access-2tbbs\") pod \"cluster-monitoring-operator-548b489cb8-zt4lx\" (UID: \"90ebb0ff-2b60-4c52-97cb-1cf589c21b61\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx" Jul 29 00:58:29.549861 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.549701 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-metrics-certs\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:29.549861 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.549749 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncmwz\" (UniqueName: \"kubernetes.io/projected/5b416fba-c1d2-4013-a4f2-a62ee1f41262-kube-api-access-ncmwz\") pod \"cluster-samples-operator-6748489656-wb72f\" (UID: \"5b416fba-c1d2-4013-a4f2-a62ee1f41262\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wb72f" Jul 29 00:58:29.549861 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.549785 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-zt4lx\" (UID: \"90ebb0ff-2b60-4c52-97cb-1cf589c21b61\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx" Jul 29 00:58:29.549861 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.549833 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6ed4d34-e21f-445c-850b-2ee2e32aa6f0-serving-cert\") pod \"console-operator-584dcb6b54-dl64r\" (UID: \"c6ed4d34-e21f-445c-850b-2ee2e32aa6f0\") " pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" Jul 29 00:58:29.550102 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.549881 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snjvk\" (UniqueName: \"kubernetes.io/projected/d3f4c767-973d-46ca-b0d9-baf948f97b4e-kube-api-access-snjvk\") pod \"network-check-source-7d88cb55d8-vvwkz\" (UID: \"d3f4c767-973d-46ca-b0d9-baf948f97b4e\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vvwkz" Jul 29 00:58:29.550102 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.549935 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b416fba-c1d2-4013-a4f2-a62ee1f41262-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wb72f\" (UID: \"5b416fba-c1d2-4013-a4f2-a62ee1f41262\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wb72f" Jul 29 00:58:29.550102 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.549967 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20eff4d6-fdde-4bb9-a489-80daad0e2321-config\") pod \"service-ca-operator-5d6c75f84f-v9h56\" (UID: \"20eff4d6-fdde-4bb9-a489-80daad0e2321\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-v9h56" Jul 29 00:58:29.550102 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.550017 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20eff4d6-fdde-4bb9-a489-80daad0e2321-serving-cert\") pod \"service-ca-operator-5d6c75f84f-v9h56\" (UID: \"20eff4d6-fdde-4bb9-a489-80daad0e2321\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-v9h56" Jul 29 00:58:29.550102 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.550056 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-stats-auth\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:29.550340 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.550217 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6ed4d34-e21f-445c-850b-2ee2e32aa6f0-config\") pod \"console-operator-584dcb6b54-dl64r\" (UID: \"c6ed4d34-e21f-445c-850b-2ee2e32aa6f0\") " pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" Jul 29 00:58:29.550442 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.550412 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/842fabe6-40df-4c93-bba3-568afdaa041e-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq\" (UID: \"842fabe6-40df-4c93-bba3-568afdaa041e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq" Jul 29 00:58:29.550871 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.550849 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6ed4d34-e21f-445c-850b-2ee2e32aa6f0-trusted-ca\") pod \"console-operator-584dcb6b54-dl64r\" (UID: \"c6ed4d34-e21f-445c-850b-2ee2e32aa6f0\") " pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" Jul 29 00:58:29.550871 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.550866 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6ed4d34-e21f-445c-850b-2ee2e32aa6f0-config\") pod \"console-operator-584dcb6b54-dl64r\" (UID: \"c6ed4d34-e21f-445c-850b-2ee2e32aa6f0\") " pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" Jul 29 00:58:29.551911 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.551890 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/842fabe6-40df-4c93-bba3-568afdaa041e-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq\" (UID: \"842fabe6-40df-4c93-bba3-568afdaa041e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq" Jul 29 00:58:29.552159 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.552145 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20eff4d6-fdde-4bb9-a489-80daad0e2321-serving-cert\") pod \"service-ca-operator-5d6c75f84f-v9h56\" (UID: \"20eff4d6-fdde-4bb9-a489-80daad0e2321\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-v9h56" Jul 29 00:58:29.552264 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.552247 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6ed4d34-e21f-445c-850b-2ee2e32aa6f0-serving-cert\") pod \"console-operator-584dcb6b54-dl64r\" (UID: \"c6ed4d34-e21f-445c-850b-2ee2e32aa6f0\") " pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" Jul 29 00:58:29.562858 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.562810 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpvl9\" (UniqueName: \"kubernetes.io/projected/842fabe6-40df-4c93-bba3-568afdaa041e-kube-api-access-cpvl9\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq\" (UID: \"842fabe6-40df-4c93-bba3-568afdaa041e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq" Jul 29 00:58:29.562945 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.562873 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qt8z\" (UniqueName: \"kubernetes.io/projected/c6ed4d34-e21f-445c-850b-2ee2e32aa6f0-kube-api-access-6qt8z\") pod \"console-operator-584dcb6b54-dl64r\" (UID: \"c6ed4d34-e21f-445c-850b-2ee2e32aa6f0\") " pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" Jul 29 00:58:29.562945 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.562903 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snjvk\" (UniqueName: \"kubernetes.io/projected/d3f4c767-973d-46ca-b0d9-baf948f97b4e-kube-api-access-snjvk\") pod \"network-check-source-7d88cb55d8-vvwkz\" (UID: \"d3f4c767-973d-46ca-b0d9-baf948f97b4e\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vvwkz" Jul 29 00:58:29.563639 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.563623 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq699\" (UniqueName: \"kubernetes.io/projected/20eff4d6-fdde-4bb9-a489-80daad0e2321-kube-api-access-fq699\") pod \"service-ca-operator-5d6c75f84f-v9h56\" (UID: \"20eff4d6-fdde-4bb9-a489-80daad0e2321\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-v9h56" Jul 29 00:58:29.651605 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.651574 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79skp\" (UniqueName: \"kubernetes.io/projected/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-kube-api-access-79skp\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:29.651605 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.651609 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-zt4lx\" (UID: \"90ebb0ff-2b60-4c52-97cb-1cf589c21b61\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx" Jul 29 00:58:29.651864 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.651649 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tbbs\" (UniqueName: \"kubernetes.io/projected/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-kube-api-access-2tbbs\") pod \"cluster-monitoring-operator-548b489cb8-zt4lx\" (UID: \"90ebb0ff-2b60-4c52-97cb-1cf589c21b61\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx" Jul 29 00:58:29.651864 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.651677 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-metrics-certs\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:29.651864 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.651715 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmwz\" (UniqueName: \"kubernetes.io/projected/5b416fba-c1d2-4013-a4f2-a62ee1f41262-kube-api-access-ncmwz\") pod \"cluster-samples-operator-6748489656-wb72f\" (UID: \"5b416fba-c1d2-4013-a4f2-a62ee1f41262\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wb72f" Jul 29 00:58:29.651864 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.651740 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-zt4lx\" (UID: \"90ebb0ff-2b60-4c52-97cb-1cf589c21b61\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx" Jul 29 00:58:29.651864 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:29.651773 2543 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 00:58:29.651864 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:29.651851 2543 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 00:58:29.652183 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:29.651877 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-cluster-monitoring-operator-tls podName:90ebb0ff-2b60-4c52-97cb-1cf589c21b61 nodeName:}" failed. No retries permitted until 2026-07-29 00:58:30.1518552 +0000 UTC m=+260.422499501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-zt4lx" (UID: "90ebb0ff-2b60-4c52-97cb-1cf589c21b61") : secret "cluster-monitoring-operator-tls" not found Jul 29 00:58:29.652183 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:29.651917 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-metrics-certs podName:e5dfce0d-d035-4da9-92bc-c6247d5b48c8 nodeName:}" failed. No retries permitted until 2026-07-29 00:58:30.151901587 +0000 UTC m=+260.422545898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-metrics-certs") pod "router-default-7d6d7bcf7c-62rxf" (UID: "e5dfce0d-d035-4da9-92bc-c6247d5b48c8") : secret "router-metrics-certs-default" not found Jul 29 00:58:29.652183 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.651940 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b416fba-c1d2-4013-a4f2-a62ee1f41262-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wb72f\" (UID: \"5b416fba-c1d2-4013-a4f2-a62ee1f41262\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wb72f" Jul 29 00:58:29.652183 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.651976 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-stats-auth\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:29.652183 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:29.652024 2543 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 00:58:29.652183 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.652040 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-service-ca-bundle\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:29.652183 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.652069 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-default-certificate\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:29.652183 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:29.652082 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b416fba-c1d2-4013-a4f2-a62ee1f41262-samples-operator-tls podName:5b416fba-c1d2-4013-a4f2-a62ee1f41262 nodeName:}" failed. No retries permitted until 2026-07-29 00:58:30.152067674 +0000 UTC m=+260.422711974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5b416fba-c1d2-4013-a4f2-a62ee1f41262-samples-operator-tls") pod "cluster-samples-operator-6748489656-wb72f" (UID: "5b416fba-c1d2-4013-a4f2-a62ee1f41262") : secret "samples-operator-tls" not found Jul 29 00:58:29.652183 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:29.652155 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-service-ca-bundle podName:e5dfce0d-d035-4da9-92bc-c6247d5b48c8 nodeName:}" failed. No retries permitted until 2026-07-29 00:58:30.152144563 +0000 UTC m=+260.422788871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-service-ca-bundle") pod "router-default-7d6d7bcf7c-62rxf" (UID: "e5dfce0d-d035-4da9-92bc-c6247d5b48c8") : configmap references non-existent config key: service-ca.crt Jul 29 00:58:29.653108 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.653086 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-zt4lx\" (UID: \"90ebb0ff-2b60-4c52-97cb-1cf589c21b61\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx" Jul 29 00:58:29.654194 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.654176 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-default-certificate\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:29.654266 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.654252 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-stats-auth\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:29.660718 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.660690 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncmwz\" (UniqueName: \"kubernetes.io/projected/5b416fba-c1d2-4013-a4f2-a62ee1f41262-kube-api-access-ncmwz\") pod \"cluster-samples-operator-6748489656-wb72f\" (UID: \"5b416fba-c1d2-4013-a4f2-a62ee1f41262\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wb72f" Jul 29 00:58:29.661614 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.661591 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tbbs\" (UniqueName: \"kubernetes.io/projected/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-kube-api-access-2tbbs\") pod \"cluster-monitoring-operator-548b489cb8-zt4lx\" (UID: \"90ebb0ff-2b60-4c52-97cb-1cf589c21b61\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx" Jul 29 00:58:29.661923 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.661908 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79skp\" (UniqueName: \"kubernetes.io/projected/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-kube-api-access-79skp\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:29.736871 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.736792 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-v9h56" Jul 29 00:58:29.745742 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.745713 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vvwkz" Jul 29 00:58:29.751560 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.751538 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq" Jul 29 00:58:29.757142 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.757118 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" Jul 29 00:58:29.888730 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.888695 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-v9h56"] Jul 29 00:58:29.891966 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:58:29.891919 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20eff4d6_fdde_4bb9_a489_80daad0e2321.slice/crio-70da00239a9a1a2dbd31cd1ba89bd9b07b320619d1260114b06861998643707d WatchSource:0}: Error finding container 70da00239a9a1a2dbd31cd1ba89bd9b07b320619d1260114b06861998643707d: Status 404 returned error can't find the container with id 70da00239a9a1a2dbd31cd1ba89bd9b07b320619d1260114b06861998643707d Jul 29 00:58:29.935413 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:29.935380 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-dl64r"] Jul 29 00:58:29.938951 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:58:29.938924 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ed4d34_e21f_445c_850b_2ee2e32aa6f0.slice/crio-7cbac3ddad44bef770ee285e1eecd66cd284cd926b041f4c864e5d9a36793b17 WatchSource:0}: Error finding container 7cbac3ddad44bef770ee285e1eecd66cd284cd926b041f4c864e5d9a36793b17: Status 404 returned error can't find the container with id 7cbac3ddad44bef770ee285e1eecd66cd284cd926b041f4c864e5d9a36793b17 Jul 29 00:58:30.110761 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:30.110689 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq"] Jul 29 00:58:30.113683 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:58:30.113655 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842fabe6_40df_4c93_bba3_568afdaa041e.slice/crio-afab7058e41d555e165bc8fe889e074f9de4ce200db63212f6dbae5f01d16073 WatchSource:0}: Error finding container afab7058e41d555e165bc8fe889e074f9de4ce200db63212f6dbae5f01d16073: Status 404 returned error can't find the container with id afab7058e41d555e165bc8fe889e074f9de4ce200db63212f6dbae5f01d16073 Jul 29 00:58:30.114179 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:30.114152 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-vvwkz"] Jul 29 00:58:30.116704 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:58:30.116679 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f4c767_973d_46ca_b0d9_baf948f97b4e.slice/crio-29820185dcc2c6fc28a97410d83a11fdc2cf1a14b37652286c71c4a2019bdf2f WatchSource:0}: Error finding container 29820185dcc2c6fc28a97410d83a11fdc2cf1a14b37652286c71c4a2019bdf2f: Status 404 returned error can't find the container with id 29820185dcc2c6fc28a97410d83a11fdc2cf1a14b37652286c71c4a2019bdf2f Jul 29 00:58:30.156347 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:30.156323 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b416fba-c1d2-4013-a4f2-a62ee1f41262-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wb72f\" (UID: \"5b416fba-c1d2-4013-a4f2-a62ee1f41262\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wb72f" Jul 29 00:58:30.156438 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:30.156371 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-service-ca-bundle\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:30.156438 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:30.156398 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-zt4lx\" (UID: \"90ebb0ff-2b60-4c52-97cb-1cf589c21b61\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx" Jul 29 00:58:30.156438 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:30.156426 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-metrics-certs\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:30.156537 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:30.156467 2543 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 00:58:30.156537 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:30.156525 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b416fba-c1d2-4013-a4f2-a62ee1f41262-samples-operator-tls podName:5b416fba-c1d2-4013-a4f2-a62ee1f41262 nodeName:}" failed. No retries permitted until 2026-07-29 00:58:31.156511654 +0000 UTC m=+261.427155938 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5b416fba-c1d2-4013-a4f2-a62ee1f41262-samples-operator-tls") pod "cluster-samples-operator-6748489656-wb72f" (UID: "5b416fba-c1d2-4013-a4f2-a62ee1f41262") : secret "samples-operator-tls" not found Jul 29 00:58:30.156602 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:30.156547 2543 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 00:58:30.156602 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:30.156584 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-metrics-certs podName:e5dfce0d-d035-4da9-92bc-c6247d5b48c8 nodeName:}" failed. No retries permitted until 2026-07-29 00:58:31.156572858 +0000 UTC m=+261.427217168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-metrics-certs") pod "router-default-7d6d7bcf7c-62rxf" (UID: "e5dfce0d-d035-4da9-92bc-c6247d5b48c8") : secret "router-metrics-certs-default" not found Jul 29 00:58:30.156665 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:30.156618 2543 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 00:58:30.156665 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:30.156659 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-cluster-monitoring-operator-tls podName:90ebb0ff-2b60-4c52-97cb-1cf589c21b61 nodeName:}" failed. No retries permitted until 2026-07-29 00:58:31.156646755 +0000 UTC m=+261.427291056 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-zt4lx" (UID: "90ebb0ff-2b60-4c52-97cb-1cf589c21b61") : secret "cluster-monitoring-operator-tls" not found Jul 29 00:58:30.156730 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:30.156702 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-service-ca-bundle podName:e5dfce0d-d035-4da9-92bc-c6247d5b48c8 nodeName:}" failed. No retries permitted until 2026-07-29 00:58:31.156687258 +0000 UTC m=+261.427331546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-service-ca-bundle") pod "router-default-7d6d7bcf7c-62rxf" (UID: "e5dfce0d-d035-4da9-92bc-c6247d5b48c8") : configmap references non-existent config key: service-ca.crt Jul 29 00:58:30.166721 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:30.166669 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" event={"ID":"c6ed4d34-e21f-445c-850b-2ee2e32aa6f0","Type":"ContainerStarted","Data":"7cbac3ddad44bef770ee285e1eecd66cd284cd926b041f4c864e5d9a36793b17"} Jul 29 00:58:30.167690 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:30.167667 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq" event={"ID":"842fabe6-40df-4c93-bba3-568afdaa041e","Type":"ContainerStarted","Data":"afab7058e41d555e165bc8fe889e074f9de4ce200db63212f6dbae5f01d16073"} Jul 29 00:58:30.168735 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:30.168715 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-v9h56" event={"ID":"20eff4d6-fdde-4bb9-a489-80daad0e2321","Type":"ContainerStarted","Data":"70da00239a9a1a2dbd31cd1ba89bd9b07b320619d1260114b06861998643707d"} Jul 29 00:58:30.170185 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:30.170163 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vvwkz" event={"ID":"d3f4c767-973d-46ca-b0d9-baf948f97b4e","Type":"ContainerStarted","Data":"06adda76a78e6ff4aadd52fa566bb4e1b61587bac317a71d08d804c1940d6d8e"} Jul 29 00:58:30.170266 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:30.170199 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vvwkz" event={"ID":"d3f4c767-973d-46ca-b0d9-baf948f97b4e","Type":"ContainerStarted","Data":"29820185dcc2c6fc28a97410d83a11fdc2cf1a14b37652286c71c4a2019bdf2f"} Jul 29 00:58:30.185139 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:30.185097 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-vvwkz" podStartSLOduration=1.18508302 podStartE2EDuration="1.18508302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:58:30.184679493 +0000 UTC m=+260.455323820" watchObservedRunningTime="2026-07-29 00:58:30.18508302 +0000 UTC m=+260.455727369" Jul 29 00:58:31.164630 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:31.164581 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-metrics-certs\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:31.164630 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:31.164674 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b416fba-c1d2-4013-a4f2-a62ee1f41262-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wb72f\" (UID: \"5b416fba-c1d2-4013-a4f2-a62ee1f41262\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wb72f" Jul 29 00:58:31.165306 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:31.164736 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-service-ca-bundle\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:31.165306 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:31.164775 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-zt4lx\" (UID: \"90ebb0ff-2b60-4c52-97cb-1cf589c21b61\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx" Jul 29 00:58:31.165306 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:31.164951 2543 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 00:58:31.165306 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:31.165015 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b416fba-c1d2-4013-a4f2-a62ee1f41262-samples-operator-tls podName:5b416fba-c1d2-4013-a4f2-a62ee1f41262 nodeName:}" failed. No retries permitted until 2026-07-29 00:58:33.164996189 +0000 UTC m=+263.435640480 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5b416fba-c1d2-4013-a4f2-a62ee1f41262-samples-operator-tls") pod "cluster-samples-operator-6748489656-wb72f" (UID: "5b416fba-c1d2-4013-a4f2-a62ee1f41262") : secret "samples-operator-tls" not found Jul 29 00:58:31.165527 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:31.165403 2543 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 00:58:31.165527 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:31.165448 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-metrics-certs podName:e5dfce0d-d035-4da9-92bc-c6247d5b48c8 nodeName:}" failed. No retries permitted until 2026-07-29 00:58:33.165432277 +0000 UTC m=+263.436076567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-metrics-certs") pod "router-default-7d6d7bcf7c-62rxf" (UID: "e5dfce0d-d035-4da9-92bc-c6247d5b48c8") : secret "router-metrics-certs-default" not found Jul 29 00:58:31.165527 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:31.165521 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-service-ca-bundle podName:e5dfce0d-d035-4da9-92bc-c6247d5b48c8 nodeName:}" failed. No retries permitted until 2026-07-29 00:58:33.165510537 +0000 UTC m=+263.436154827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-service-ca-bundle") pod "router-default-7d6d7bcf7c-62rxf" (UID: "e5dfce0d-d035-4da9-92bc-c6247d5b48c8") : configmap references non-existent config key: service-ca.crt Jul 29 00:58:31.165684 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:31.165617 2543 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 00:58:31.165684 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:31.165653 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-cluster-monitoring-operator-tls podName:90ebb0ff-2b60-4c52-97cb-1cf589c21b61 nodeName:}" failed. No retries permitted until 2026-07-29 00:58:33.165641021 +0000 UTC m=+263.436285311 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-zt4lx" (UID: "90ebb0ff-2b60-4c52-97cb-1cf589c21b61") : secret "cluster-monitoring-operator-tls" not found Jul 29 00:58:33.181146 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.181072 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/0.log" Jul 29 00:58:33.181146 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.181118 2543 generic.go:356] "Generic (PLEG): container finished" podID="c6ed4d34-e21f-445c-850b-2ee2e32aa6f0" containerID="962ab94b002a6299b38a5adfecc2b59b4ee6cc5438a169c9055bc7b6d902bafb" exitCode=255 Jul 29 00:58:33.181604 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.181210 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" event={"ID":"c6ed4d34-e21f-445c-850b-2ee2e32aa6f0","Type":"ContainerDied","Data":"962ab94b002a6299b38a5adfecc2b59b4ee6cc5438a169c9055bc7b6d902bafb"} Jul 29 00:58:33.181604 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.181424 2543 scope.go:117] "RemoveContainer" containerID="962ab94b002a6299b38a5adfecc2b59b4ee6cc5438a169c9055bc7b6d902bafb" Jul 29 00:58:33.182661 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.182639 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq" event={"ID":"842fabe6-40df-4c93-bba3-568afdaa041e","Type":"ContainerStarted","Data":"da916dceff0599eaadd8e10e7051ad41795d107ce025f3dde706c78fb98814b6"} Jul 29 00:58:33.184118 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.184098 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-v9h56" event={"ID":"20eff4d6-fdde-4bb9-a489-80daad0e2321","Type":"ContainerStarted","Data":"224550b704a18a7b19c6bfea72cd75594390027d2aab28527ef7afada6738ff8"} Jul 29 00:58:33.184371 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.184354 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-service-ca-bundle\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:33.184432 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.184390 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-zt4lx\" (UID: \"90ebb0ff-2b60-4c52-97cb-1cf589c21b61\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx" Jul 29 00:58:33.184432 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.184420 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-metrics-certs\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:33.184533 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.184471 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b416fba-c1d2-4013-a4f2-a62ee1f41262-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wb72f\" (UID: \"5b416fba-c1d2-4013-a4f2-a62ee1f41262\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wb72f" Jul 29 00:58:33.184533 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:33.184511 2543 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 00:58:33.184630 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:33.184533 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-service-ca-bundle podName:e5dfce0d-d035-4da9-92bc-c6247d5b48c8 nodeName:}" failed. No retries permitted until 2026-07-29 00:58:37.184512845 +0000 UTC m=+267.455157134 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-service-ca-bundle") pod "router-default-7d6d7bcf7c-62rxf" (UID: "e5dfce0d-d035-4da9-92bc-c6247d5b48c8") : configmap references non-existent config key: service-ca.crt Jul 29 00:58:33.184630 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:33.184562 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-cluster-monitoring-operator-tls podName:90ebb0ff-2b60-4c52-97cb-1cf589c21b61 nodeName:}" failed. No retries permitted until 2026-07-29 00:58:37.184546756 +0000 UTC m=+267.455191044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-zt4lx" (UID: "90ebb0ff-2b60-4c52-97cb-1cf589c21b61") : secret "cluster-monitoring-operator-tls" not found Jul 29 00:58:33.184630 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:33.184566 2543 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 00:58:33.184630 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:33.184608 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-metrics-certs podName:e5dfce0d-d035-4da9-92bc-c6247d5b48c8 nodeName:}" failed. No retries permitted until 2026-07-29 00:58:37.184594193 +0000 UTC m=+267.455238478 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-metrics-certs") pod "router-default-7d6d7bcf7c-62rxf" (UID: "e5dfce0d-d035-4da9-92bc-c6247d5b48c8") : secret "router-metrics-certs-default" not found Jul 29 00:58:33.184630 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:33.184566 2543 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 00:58:33.184915 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:33.184653 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b416fba-c1d2-4013-a4f2-a62ee1f41262-samples-operator-tls podName:5b416fba-c1d2-4013-a4f2-a62ee1f41262 nodeName:}" failed. No retries permitted until 2026-07-29 00:58:37.184643513 +0000 UTC m=+267.455287798 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5b416fba-c1d2-4013-a4f2-a62ee1f41262-samples-operator-tls") pod "cluster-samples-operator-6748489656-wb72f" (UID: "5b416fba-c1d2-4013-a4f2-a62ee1f41262") : secret "samples-operator-tls" not found Jul 29 00:58:33.209965 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.209924 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq" podStartSLOduration=1.407915568 podStartE2EDuration="4.20991034s" totalImagesPullingTime="2.801994772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:58:33.208758771 +0000 UTC m=+263.479403078" watchObservedRunningTime="2026-07-29 00:58:33.20991034 +0000 UTC m=+263.480554647" Jul 29 00:58:33.223351 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.223304 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-v9h56" podStartSLOduration=1.203966446 podStartE2EDuration="4.223286878s" totalImagesPullingTime="3.019320432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:58:33.221989027 +0000 UTC m=+263.492633335" watchObservedRunningTime="2026-07-29 00:58:33.223286878 +0000 UTC m=+263.493931186" Jul 29 00:58:33.554112 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.554035 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ppxd8"] Jul 29 00:58:33.556116 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.556099 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:33.558404 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.558382 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 29 00:58:33.558702 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.558670 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 29 00:58:33.558792 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.558780 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 29 00:58:33.559134 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.559118 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 29 00:58:33.559225 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.559152 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vj9dt\"" Jul 29 00:58:33.571864 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.571835 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ppxd8"] Jul 29 00:58:33.588571 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.588538 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e12d3020-b8bc-478a-979e-db748500760e-crio-socket\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:33.588716 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.588689 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e12d3020-b8bc-478a-979e-db748500760e-data-volume\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:33.588846 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.588804 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jh8w\" (UniqueName: \"kubernetes.io/projected/e12d3020-b8bc-478a-979e-db748500760e-kube-api-access-6jh8w\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:33.588961 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.588883 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e12d3020-b8bc-478a-979e-db748500760e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:33.588961 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.588915 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e12d3020-b8bc-478a-979e-db748500760e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:33.689965 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.689928 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e12d3020-b8bc-478a-979e-db748500760e-data-volume\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:33.690145 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.690006 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jh8w\" (UniqueName: \"kubernetes.io/projected/e12d3020-b8bc-478a-979e-db748500760e-kube-api-access-6jh8w\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:33.690145 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.690034 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e12d3020-b8bc-478a-979e-db748500760e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:33.690145 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.690054 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e12d3020-b8bc-478a-979e-db748500760e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:33.690145 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.690081 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e12d3020-b8bc-478a-979e-db748500760e-crio-socket\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:33.690355 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:33.690178 2543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 00:58:33.690355 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.690193 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e12d3020-b8bc-478a-979e-db748500760e-crio-socket\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:33.690355 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:33.690259 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12d3020-b8bc-478a-979e-db748500760e-insights-runtime-extractor-tls podName:e12d3020-b8bc-478a-979e-db748500760e nodeName:}" failed. No retries permitted until 2026-07-29 00:58:34.190236193 +0000 UTC m=+264.460880498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e12d3020-b8bc-478a-979e-db748500760e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ppxd8" (UID: "e12d3020-b8bc-478a-979e-db748500760e") : secret "insights-runtime-extractor-tls" not found Jul 29 00:58:33.690355 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:33.690183 2543 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 00:58:33.690355 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:33.690329 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e12d3020-b8bc-478a-979e-db748500760e-kube-rbac-proxy-cm podName:e12d3020-b8bc-478a-979e-db748500760e nodeName:}" failed. No retries permitted until 2026-07-29 00:58:34.190312952 +0000 UTC m=+264.460957251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e12d3020-b8bc-478a-979e-db748500760e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ppxd8" (UID: "e12d3020-b8bc-478a-979e-db748500760e") : configmap "kube-rbac-proxy" not found Jul 29 00:58:33.690520 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.690400 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e12d3020-b8bc-478a-979e-db748500760e-data-volume\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:33.698062 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:33.698040 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jh8w\" (UniqueName: \"kubernetes.io/projected/e12d3020-b8bc-478a-979e-db748500760e-kube-api-access-6jh8w\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:34.188930 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:34.188899 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/1.log" Jul 29 00:58:34.189346 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:34.189331 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/0.log" Jul 29 00:58:34.189393 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:34.189365 2543 generic.go:356] "Generic (PLEG): container finished" podID="c6ed4d34-e21f-445c-850b-2ee2e32aa6f0" containerID="a06f72fd7a5b7937fef2f9492504c980720067907e6bd5d28674fad1d0fc3903" exitCode=255 Jul 29 00:58:34.189469 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:34.189443 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" event={"ID":"c6ed4d34-e21f-445c-850b-2ee2e32aa6f0","Type":"ContainerDied","Data":"a06f72fd7a5b7937fef2f9492504c980720067907e6bd5d28674fad1d0fc3903"} Jul 29 00:58:34.189576 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:34.189491 2543 scope.go:117] "RemoveContainer" containerID="962ab94b002a6299b38a5adfecc2b59b4ee6cc5438a169c9055bc7b6d902bafb" Jul 29 00:58:34.189720 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:34.189694 2543 scope.go:117] "RemoveContainer" containerID="a06f72fd7a5b7937fef2f9492504c980720067907e6bd5d28674fad1d0fc3903" Jul 29 00:58:34.189928 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:34.189890 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-dl64r_openshift-console-operator(c6ed4d34-e21f-445c-850b-2ee2e32aa6f0)\"" pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" podUID="c6ed4d34-e21f-445c-850b-2ee2e32aa6f0" Jul 29 00:58:34.195145 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:34.195121 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e12d3020-b8bc-478a-979e-db748500760e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:34.195235 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:34.195170 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e12d3020-b8bc-478a-979e-db748500760e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:34.195304 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:34.195283 2543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 00:58:34.195372 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:34.195324 2543 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 00:58:34.195372 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:34.195349 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12d3020-b8bc-478a-979e-db748500760e-insights-runtime-extractor-tls podName:e12d3020-b8bc-478a-979e-db748500760e nodeName:}" failed. No retries permitted until 2026-07-29 00:58:35.195332033 +0000 UTC m=+265.465976339 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e12d3020-b8bc-478a-979e-db748500760e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ppxd8" (UID: "e12d3020-b8bc-478a-979e-db748500760e") : secret "insights-runtime-extractor-tls" not found Jul 29 00:58:34.195372 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:34.195368 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e12d3020-b8bc-478a-979e-db748500760e-kube-rbac-proxy-cm podName:e12d3020-b8bc-478a-979e-db748500760e nodeName:}" failed. No retries permitted until 2026-07-29 00:58:35.195357851 +0000 UTC m=+265.466002139 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e12d3020-b8bc-478a-979e-db748500760e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ppxd8" (UID: "e12d3020-b8bc-478a-979e-db748500760e") : configmap "kube-rbac-proxy" not found Jul 29 00:58:34.640150 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:34.640110 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-q9nqv"] Jul 29 00:58:34.642833 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:34.642795 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-q9nqv" Jul 29 00:58:34.644980 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:34.644958 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 29 00:58:34.645069 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:34.644958 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8sn9k\"" Jul 29 00:58:34.645899 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:34.645879 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 29 00:58:34.652174 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:34.652154 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-q9nqv"] Jul 29 00:58:34.700005 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:34.699977 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z88wr\" (UniqueName: \"kubernetes.io/projected/2d05aed8-edd7-4d87-8c99-f621979c8178-kube-api-access-z88wr\") pod \"migrator-6865c9967-q9nqv\" (UID: \"2d05aed8-edd7-4d87-8c99-f621979c8178\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-q9nqv" Jul 29 00:58:34.800984 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:34.800953 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z88wr\" (UniqueName: \"kubernetes.io/projected/2d05aed8-edd7-4d87-8c99-f621979c8178-kube-api-access-z88wr\") pod \"migrator-6865c9967-q9nqv\" (UID: \"2d05aed8-edd7-4d87-8c99-f621979c8178\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-q9nqv" Jul 29 00:58:34.807909 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:34.807888 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z88wr\" (UniqueName: \"kubernetes.io/projected/2d05aed8-edd7-4d87-8c99-f621979c8178-kube-api-access-z88wr\") pod \"migrator-6865c9967-q9nqv\" (UID: \"2d05aed8-edd7-4d87-8c99-f621979c8178\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-q9nqv" Jul 29 00:58:34.952872 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:34.952793 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-q9nqv" Jul 29 00:58:35.072061 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:35.072012 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-q9nqv"] Jul 29 00:58:35.073989 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:58:35.073961 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d05aed8_edd7_4d87_8c99_f621979c8178.slice/crio-2f387d55f2d3b0b17c49c913cd83e0a6f7edfcb49dea0242738c6dcd5f9ba71f WatchSource:0}: Error finding container 2f387d55f2d3b0b17c49c913cd83e0a6f7edfcb49dea0242738c6dcd5f9ba71f: Status 404 returned error can't find the container with id 2f387d55f2d3b0b17c49c913cd83e0a6f7edfcb49dea0242738c6dcd5f9ba71f Jul 29 00:58:35.193825 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:35.193786 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-q9nqv" event={"ID":"2d05aed8-edd7-4d87-8c99-f621979c8178","Type":"ContainerStarted","Data":"2f387d55f2d3b0b17c49c913cd83e0a6f7edfcb49dea0242738c6dcd5f9ba71f"} Jul 29 00:58:35.195176 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:35.195147 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/1.log" Jul 29 00:58:35.195469 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:35.195453 2543 scope.go:117] "RemoveContainer" containerID="a06f72fd7a5b7937fef2f9492504c980720067907e6bd5d28674fad1d0fc3903" Jul 29 00:58:35.195609 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:35.195592 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-dl64r_openshift-console-operator(c6ed4d34-e21f-445c-850b-2ee2e32aa6f0)\"" pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" podUID="c6ed4d34-e21f-445c-850b-2ee2e32aa6f0" Jul 29 00:58:35.203606 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:35.203564 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e12d3020-b8bc-478a-979e-db748500760e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:35.203686 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:35.203672 2543 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 00:58:35.203686 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:35.203682 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e12d3020-b8bc-478a-979e-db748500760e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:35.203762 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:35.203719 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e12d3020-b8bc-478a-979e-db748500760e-kube-rbac-proxy-cm podName:e12d3020-b8bc-478a-979e-db748500760e nodeName:}" failed. No retries permitted until 2026-07-29 00:58:37.203706327 +0000 UTC m=+267.474350612 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e12d3020-b8bc-478a-979e-db748500760e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ppxd8" (UID: "e12d3020-b8bc-478a-979e-db748500760e") : configmap "kube-rbac-proxy" not found Jul 29 00:58:35.203800 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:35.203763 2543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 00:58:35.203861 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:35.203801 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12d3020-b8bc-478a-979e-db748500760e-insights-runtime-extractor-tls podName:e12d3020-b8bc-478a-979e-db748500760e nodeName:}" failed. No retries permitted until 2026-07-29 00:58:37.203790353 +0000 UTC m=+267.474434639 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e12d3020-b8bc-478a-979e-db748500760e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ppxd8" (UID: "e12d3020-b8bc-478a-979e-db748500760e") : secret "insights-runtime-extractor-tls" not found Jul 29 00:58:35.546615 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:35.546545 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-thq5l_5719a514-b8de-4661-ab2a-fe0587726ae8/dns-node-resolver/0.log" Jul 29 00:58:36.546377 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:36.546343 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nqv9s_a2127c7c-06fc-4849-ab98-2975c0862540/node-ca/0.log" Jul 29 00:58:37.147677 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:37.147596 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h6ncm_9ab87c7b-6a40-4383-a3cb-90f40e463040/serve-healthcheck-canary/0.log" Jul 29 00:58:37.204279 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:37.204249 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-q9nqv" event={"ID":"2d05aed8-edd7-4d87-8c99-f621979c8178","Type":"ContainerStarted","Data":"73f9794aec19161841f5990b05f893aa9ebbb55a676cc1df94adc27730d535b4"} Jul 29 00:58:37.204400 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:37.204301 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-q9nqv" event={"ID":"2d05aed8-edd7-4d87-8c99-f621979c8178","Type":"ContainerStarted","Data":"96241f0cd139dbee756bb3e7d8bf5ca740f0c9ac4c03a96fdba6d7884a120860"} Jul 29 00:58:37.219738 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:37.219701 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-q9nqv" podStartSLOduration=1.404484134 podStartE2EDuration="3.219687432s" totalImagesPullingTime="1.815203298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:58:37.219084613 +0000 UTC m=+267.489728920" watchObservedRunningTime="2026-07-29 00:58:37.219687432 +0000 UTC m=+267.490331767" Jul 29 00:58:37.220685 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:37.220666 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-metrics-certs\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:37.220745 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:37.220732 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b416fba-c1d2-4013-a4f2-a62ee1f41262-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wb72f\" (UID: \"5b416fba-c1d2-4013-a4f2-a62ee1f41262\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wb72f" Jul 29 00:58:37.220801 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:37.220786 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e12d3020-b8bc-478a-979e-db748500760e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:37.220859 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:37.220807 2543 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 00:58:37.220859 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:37.220835 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e12d3020-b8bc-478a-979e-db748500760e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:37.220932 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:37.220873 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-metrics-certs podName:e5dfce0d-d035-4da9-92bc-c6247d5b48c8 nodeName:}" failed. No retries permitted until 2026-07-29 00:58:45.220858034 +0000 UTC m=+275.491502319 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-metrics-certs") pod "router-default-7d6d7bcf7c-62rxf" (UID: "e5dfce0d-d035-4da9-92bc-c6247d5b48c8") : secret "router-metrics-certs-default" not found Jul 29 00:58:37.220932 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:37.220891 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-service-ca-bundle\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:37.221030 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:37.220938 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-zt4lx\" (UID: \"90ebb0ff-2b60-4c52-97cb-1cf589c21b61\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx" Jul 29 00:58:37.221030 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:37.220895 2543 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 00:58:37.221030 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:37.220989 2543 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 00:58:37.221030 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:37.220896 2543 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 00:58:37.221030 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:37.220904 2543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 00:58:37.221220 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:37.220955 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-service-ca-bundle podName:e5dfce0d-d035-4da9-92bc-c6247d5b48c8 nodeName:}" failed. No retries permitted until 2026-07-29 00:58:45.220944621 +0000 UTC m=+275.491588912 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-service-ca-bundle") pod "router-default-7d6d7bcf7c-62rxf" (UID: "e5dfce0d-d035-4da9-92bc-c6247d5b48c8") : configmap references non-existent config key: service-ca.crt Jul 29 00:58:37.221220 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:37.221088 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-cluster-monitoring-operator-tls podName:90ebb0ff-2b60-4c52-97cb-1cf589c21b61 nodeName:}" failed. No retries permitted until 2026-07-29 00:58:45.221067615 +0000 UTC m=+275.491711910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-zt4lx" (UID: "90ebb0ff-2b60-4c52-97cb-1cf589c21b61") : secret "cluster-monitoring-operator-tls" not found Jul 29 00:58:37.221220 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:37.221132 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b416fba-c1d2-4013-a4f2-a62ee1f41262-samples-operator-tls podName:5b416fba-c1d2-4013-a4f2-a62ee1f41262 nodeName:}" failed. No retries permitted until 2026-07-29 00:58:45.221102139 +0000 UTC m=+275.491746437 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5b416fba-c1d2-4013-a4f2-a62ee1f41262-samples-operator-tls") pod "cluster-samples-operator-6748489656-wb72f" (UID: "5b416fba-c1d2-4013-a4f2-a62ee1f41262") : secret "samples-operator-tls" not found Jul 29 00:58:37.221220 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:37.221157 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e12d3020-b8bc-478a-979e-db748500760e-kube-rbac-proxy-cm podName:e12d3020-b8bc-478a-979e-db748500760e nodeName:}" failed. No retries permitted until 2026-07-29 00:58:41.221146305 +0000 UTC m=+271.491790599 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e12d3020-b8bc-478a-979e-db748500760e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ppxd8" (UID: "e12d3020-b8bc-478a-979e-db748500760e") : configmap "kube-rbac-proxy" not found Jul 29 00:58:37.221220 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:37.221177 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12d3020-b8bc-478a-979e-db748500760e-insights-runtime-extractor-tls podName:e12d3020-b8bc-478a-979e-db748500760e nodeName:}" failed. No retries permitted until 2026-07-29 00:58:41.221167886 +0000 UTC m=+271.491812175 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e12d3020-b8bc-478a-979e-db748500760e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ppxd8" (UID: "e12d3020-b8bc-478a-979e-db748500760e") : secret "insights-runtime-extractor-tls" not found Jul 29 00:58:39.757809 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:39.757734 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" Jul 29 00:58:39.757809 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:39.757782 2543 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" Jul 29 00:58:39.758182 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:39.758075 2543 scope.go:117] "RemoveContainer" containerID="a06f72fd7a5b7937fef2f9492504c980720067907e6bd5d28674fad1d0fc3903" Jul 29 00:58:39.758228 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:39.758208 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-dl64r_openshift-console-operator(c6ed4d34-e21f-445c-850b-2ee2e32aa6f0)\"" pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" podUID="c6ed4d34-e21f-445c-850b-2ee2e32aa6f0" Jul 29 00:58:41.255618 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:41.255582 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e12d3020-b8bc-478a-979e-db748500760e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:41.256029 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:41.255622 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e12d3020-b8bc-478a-979e-db748500760e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:41.256029 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:41.255692 2543 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 00:58:41.256029 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:41.255720 2543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 00:58:41.256029 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:41.255757 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e12d3020-b8bc-478a-979e-db748500760e-kube-rbac-proxy-cm podName:e12d3020-b8bc-478a-979e-db748500760e nodeName:}" failed. No retries permitted until 2026-07-29 00:58:49.255738284 +0000 UTC m=+279.526382595 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e12d3020-b8bc-478a-979e-db748500760e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ppxd8" (UID: "e12d3020-b8bc-478a-979e-db748500760e") : configmap "kube-rbac-proxy" not found Jul 29 00:58:41.256029 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:41.255840 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12d3020-b8bc-478a-979e-db748500760e-insights-runtime-extractor-tls podName:e12d3020-b8bc-478a-979e-db748500760e nodeName:}" failed. No retries permitted until 2026-07-29 00:58:49.255803679 +0000 UTC m=+279.526447969 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e12d3020-b8bc-478a-979e-db748500760e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ppxd8" (UID: "e12d3020-b8bc-478a-979e-db748500760e") : secret "insights-runtime-extractor-tls" not found Jul 29 00:58:45.286871 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:45.286801 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b416fba-c1d2-4013-a4f2-a62ee1f41262-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wb72f\" (UID: \"5b416fba-c1d2-4013-a4f2-a62ee1f41262\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wb72f" Jul 29 00:58:45.287298 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:45.286989 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-service-ca-bundle\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:45.287298 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:45.287023 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-zt4lx\" (UID: \"90ebb0ff-2b60-4c52-97cb-1cf589c21b61\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx" Jul 29 00:58:45.287298 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:45.287068 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-metrics-certs\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:45.287298 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:45.287194 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-service-ca-bundle podName:e5dfce0d-d035-4da9-92bc-c6247d5b48c8 nodeName:}" failed. No retries permitted until 2026-07-29 00:59:01.287173435 +0000 UTC m=+291.557817731 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-service-ca-bundle") pod "router-default-7d6d7bcf7c-62rxf" (UID: "e5dfce0d-d035-4da9-92bc-c6247d5b48c8") : configmap references non-existent config key: service-ca.crt Jul 29 00:58:45.287298 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:45.287206 2543 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 29 00:58:45.287298 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:45.287274 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-cluster-monitoring-operator-tls podName:90ebb0ff-2b60-4c52-97cb-1cf589c21b61 nodeName:}" failed. No retries permitted until 2026-07-29 00:59:01.287258985 +0000 UTC m=+291.557903271 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-zt4lx" (UID: "90ebb0ff-2b60-4c52-97cb-1cf589c21b61") : secret "cluster-monitoring-operator-tls" not found Jul 29 00:58:45.289231 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:45.289204 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5b416fba-c1d2-4013-a4f2-a62ee1f41262-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wb72f\" (UID: \"5b416fba-c1d2-4013-a4f2-a62ee1f41262\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wb72f" Jul 29 00:58:45.289360 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:45.289342 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-metrics-certs\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:58:45.437345 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:45.437314 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wb72f" Jul 29 00:58:45.554752 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:45.554683 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wb72f"] Jul 29 00:58:46.231621 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:46.231583 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wb72f" event={"ID":"5b416fba-c1d2-4013-a4f2-a62ee1f41262","Type":"ContainerStarted","Data":"de70c862f85a900ce98191a0e8ad0852a66c9590064b4f92f9df6a6cc2ce1dd3"} Jul 29 00:58:48.239215 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:48.239180 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wb72f" event={"ID":"5b416fba-c1d2-4013-a4f2-a62ee1f41262","Type":"ContainerStarted","Data":"62bff5c7d179875579511e5d396188327b336adc6f3262e8da9e7a94c40787cf"} Jul 29 00:58:48.239605 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:48.239235 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wb72f" event={"ID":"5b416fba-c1d2-4013-a4f2-a62ee1f41262","Type":"ContainerStarted","Data":"13c7068cda0491db201916751dbce30bfd8a3dcb83283c9ec9f6ea413a07dbb0"} Jul 29 00:58:48.253645 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:48.253598 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wb72f" podStartSLOduration=17.644188983 podStartE2EDuration="19.25358465s" totalImagesPullingTime="1.609395667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:58:48.253059336 +0000 UTC m=+278.523703637" watchObservedRunningTime="2026-07-29 00:58:48.25358465 +0000 UTC m=+278.524228956" Jul 29 00:58:49.323731 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:49.323699 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e12d3020-b8bc-478a-979e-db748500760e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:49.324106 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:49.323740 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e12d3020-b8bc-478a-979e-db748500760e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:49.324106 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:49.323881 2543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 00:58:49.324106 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:49.323948 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12d3020-b8bc-478a-979e-db748500760e-insights-runtime-extractor-tls podName:e12d3020-b8bc-478a-979e-db748500760e nodeName:}" failed. No retries permitted until 2026-07-29 00:59:05.323927196 +0000 UTC m=+295.594571481 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e12d3020-b8bc-478a-979e-db748500760e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ppxd8" (UID: "e12d3020-b8bc-478a-979e-db748500760e") : secret "insights-runtime-extractor-tls" not found Jul 29 00:58:49.324450 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:49.324433 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e12d3020-b8bc-478a-979e-db748500760e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:58:51.337633 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:51.337604 2543 scope.go:117] "RemoveContainer" containerID="a06f72fd7a5b7937fef2f9492504c980720067907e6bd5d28674fad1d0fc3903" Jul 29 00:58:51.940690 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:51.940642 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" podUID="3613abf5-ad33-4253-bf63-c8f7ccd99759" Jul 29 00:58:51.940872 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:51.940641 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" podUID="b8a3faba-c784-4189-9c88-b6743126556a" Jul 29 00:58:51.940872 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:51.940642 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7jhhc" podUID="078d8065-0985-4266-b9ec-e607f7372bb7" Jul 29 00:58:52.253572 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:52.253495 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 00:58:52.253919 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:52.253905 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/1.log" Jul 29 00:58:52.253966 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:52.253936 2543 generic.go:356] "Generic (PLEG): container finished" podID="c6ed4d34-e21f-445c-850b-2ee2e32aa6f0" containerID="31af71b4261fc658a04b8eb2236d3bd3b21ce9daa5b46223cfcd2165ea8c66d6" exitCode=255 Jul 29 00:58:52.254036 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:52.254024 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:58:52.254091 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:52.254044 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" Jul 29 00:58:52.254091 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:52.254046 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" event={"ID":"c6ed4d34-e21f-445c-850b-2ee2e32aa6f0","Type":"ContainerDied","Data":"31af71b4261fc658a04b8eb2236d3bd3b21ce9daa5b46223cfcd2165ea8c66d6"} Jul 29 00:58:52.254091 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:52.254050 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7jhhc" Jul 29 00:58:52.254194 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:52.254100 2543 scope.go:117] "RemoveContainer" containerID="a06f72fd7a5b7937fef2f9492504c980720067907e6bd5d28674fad1d0fc3903" Jul 29 00:58:52.254403 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:52.254389 2543 scope.go:117] "RemoveContainer" containerID="31af71b4261fc658a04b8eb2236d3bd3b21ce9daa5b46223cfcd2165ea8c66d6" Jul 29 00:58:52.254579 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:52.254553 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-dl64r_openshift-console-operator(c6ed4d34-e21f-445c-850b-2ee2e32aa6f0)\"" pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" podUID="c6ed4d34-e21f-445c-850b-2ee2e32aa6f0" Jul 29 00:58:53.259007 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:53.258978 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 00:58:55.070771 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:55.070737 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls\") pod \"dns-default-7jhhc\" (UID: \"078d8065-0985-4266-b9ec-e607f7372bb7\") " pod="openshift-dns/dns-default-7jhhc" Jul 29 00:58:55.071274 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:55.070872 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:58:55.071274 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:55.070910 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-b7qp4\" (UID: \"b8a3faba-c784-4189-9c88-b6743126556a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" Jul 29 00:58:55.073044 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:55.073024 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/078d8065-0985-4266-b9ec-e607f7372bb7-metrics-tls\") pod \"dns-default-7jhhc\" (UID: \"078d8065-0985-4266-b9ec-e607f7372bb7\") " pod="openshift-dns/dns-default-7jhhc" Jul 29 00:58:55.073139 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:55.073094 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b8a3faba-c784-4189-9c88-b6743126556a-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-b7qp4\" (UID: \"b8a3faba-c784-4189-9c88-b6743126556a\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" Jul 29 00:58:55.073239 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:55.073219 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls\") pod \"image-registry-684c9f4678-nd9lv\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:58:55.257202 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:55.257169 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9mcp8\"" Jul 29 00:58:55.257202 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:55.257173 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6dlx2\"" Jul 29 00:58:55.257416 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:55.257273 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-bwc6w\"" Jul 29 00:58:55.265403 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:55.265159 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7jhhc" Jul 29 00:58:55.267731 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:55.267705 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" Jul 29 00:58:55.267849 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:55.267746 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:58:55.431203 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:55.431171 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-684c9f4678-nd9lv"] Jul 29 00:58:55.434515 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:58:55.434483 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3613abf5_ad33_4253_bf63_c8f7ccd99759.slice/crio-03f19e1f59a558bb3abe75aa6a94a38bd2140fa5b9b288211443bdef19e80dc0 WatchSource:0}: Error finding container 03f19e1f59a558bb3abe75aa6a94a38bd2140fa5b9b288211443bdef19e80dc0: Status 404 returned error can't find the container with id 03f19e1f59a558bb3abe75aa6a94a38bd2140fa5b9b288211443bdef19e80dc0 Jul 29 00:58:55.647934 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:55.647802 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4"] Jul 29 00:58:55.650320 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:55.650295 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7jhhc"] Jul 29 00:58:55.651066 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:58:55.650981 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a3faba_c784_4189_9c88_b6743126556a.slice/crio-d0fad5acad682c988b5b5e738742b80ec799ffa99cd96987e7ffded54171347c WatchSource:0}: Error finding container d0fad5acad682c988b5b5e738742b80ec799ffa99cd96987e7ffded54171347c: Status 404 returned error can't find the container with id d0fad5acad682c988b5b5e738742b80ec799ffa99cd96987e7ffded54171347c Jul 29 00:58:55.653061 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:58:55.653036 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078d8065_0985_4266_b9ec_e607f7372bb7.slice/crio-469ab14bfc17b860234f1e5c52d764ba031a2376418774912b22e28f9753ad9a WatchSource:0}: Error finding container 469ab14bfc17b860234f1e5c52d764ba031a2376418774912b22e28f9753ad9a: Status 404 returned error can't find the container with id 469ab14bfc17b860234f1e5c52d764ba031a2376418774912b22e28f9753ad9a Jul 29 00:58:56.269361 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:56.269323 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" event={"ID":"b8a3faba-c784-4189-9c88-b6743126556a","Type":"ContainerStarted","Data":"d0fad5acad682c988b5b5e738742b80ec799ffa99cd96987e7ffded54171347c"} Jul 29 00:58:56.270696 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:56.270670 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" event={"ID":"3613abf5-ad33-4253-bf63-c8f7ccd99759","Type":"ContainerStarted","Data":"ed2e36729f73fa1d246e0e4e452737a15a617b21609e318d8a7063514890f274"} Jul 29 00:58:56.270851 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:56.270711 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" event={"ID":"3613abf5-ad33-4253-bf63-c8f7ccd99759","Type":"ContainerStarted","Data":"03f19e1f59a558bb3abe75aa6a94a38bd2140fa5b9b288211443bdef19e80dc0"} Jul 29 00:58:56.270851 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:56.270733 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:58:56.271691 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:56.271671 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7jhhc" event={"ID":"078d8065-0985-4266-b9ec-e607f7372bb7","Type":"ContainerStarted","Data":"469ab14bfc17b860234f1e5c52d764ba031a2376418774912b22e28f9753ad9a"} Jul 29 00:58:56.287077 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:56.287038 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" podStartSLOduration=285.287024989 podStartE2EDuration="4m45.287024989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:58:56.286559685 +0000 UTC m=+286.557203992" watchObservedRunningTime="2026-07-29 00:58:56.287024989 +0000 UTC m=+286.557669295" Jul 29 00:58:57.276541 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:57.276469 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" event={"ID":"b8a3faba-c784-4189-9c88-b6743126556a","Type":"ContainerStarted","Data":"83290824cfc4d32b800b12f489197b769ed519b2b9c21a46131775fc88e240f1"} Jul 29 00:58:57.289514 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:57.289451 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-b7qp4" podStartSLOduration=264.781666096 podStartE2EDuration="4m26.289436707s" totalImagesPullingTime="1.507770611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:58:57.289402057 +0000 UTC m=+287.560046365" watchObservedRunningTime="2026-07-29 00:58:57.289436707 +0000 UTC m=+287.560081012" Jul 29 00:58:58.282144 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:58.282098 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7jhhc" event={"ID":"078d8065-0985-4266-b9ec-e607f7372bb7","Type":"ContainerStarted","Data":"92727af7026f3013ebadeaa18d65e4e11d29d24b62e916825f108da6e93dbaf9"} Jul 29 00:58:58.282572 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:58.282173 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7jhhc" event={"ID":"078d8065-0985-4266-b9ec-e607f7372bb7","Type":"ContainerStarted","Data":"d63fb607c1f3a99ae8012d6c4f5df5add8eb3652fdeadc9a3dfb8a2d3453250f"} Jul 29 00:58:58.303133 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:58.303073 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7jhhc" podStartSLOduration=251.284583289 podStartE2EDuration="4m13.303051904s" totalImagesPullingTime="2.018468615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:58:58.30273116 +0000 UTC m=+288.573375482" watchObservedRunningTime="2026-07-29 00:58:58.303051904 +0000 UTC m=+288.573696212" Jul 29 00:58:59.285980 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:59.285950 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7jhhc" Jul 29 00:58:59.757785 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:59.757746 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" Jul 29 00:58:59.757964 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:59.757801 2543 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" Jul 29 00:58:59.758147 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:58:59.758134 2543 scope.go:117] "RemoveContainer" containerID="31af71b4261fc658a04b8eb2236d3bd3b21ce9daa5b46223cfcd2165ea8c66d6" Jul 29 00:58:59.758298 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:58:59.758281 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-dl64r_openshift-console-operator(c6ed4d34-e21f-445c-850b-2ee2e32aa6f0)\"" pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" podUID="c6ed4d34-e21f-445c-850b-2ee2e32aa6f0" Jul 29 00:59:01.317993 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:01.317947 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-service-ca-bundle\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:59:01.318594 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:01.318007 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-zt4lx\" (UID: \"90ebb0ff-2b60-4c52-97cb-1cf589c21b61\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx" Jul 29 00:59:01.318594 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:01.318545 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5dfce0d-d035-4da9-92bc-c6247d5b48c8-service-ca-bundle\") pod \"router-default-7d6d7bcf7c-62rxf\" (UID: \"e5dfce0d-d035-4da9-92bc-c6247d5b48c8\") " pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:59:01.320311 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:01.320290 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/90ebb0ff-2b60-4c52-97cb-1cf589c21b61-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-zt4lx\" (UID: \"90ebb0ff-2b60-4c52-97cb-1cf589c21b61\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx" Jul 29 00:59:01.330203 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:01.330180 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx" Jul 29 00:59:01.343457 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:01.343438 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:59:01.470705 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:01.470672 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx"] Jul 29 00:59:01.473662 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:59:01.473634 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ebb0ff_2b60_4c52_97cb_1cf589c21b61.slice/crio-827ecbe6b3b684937f8897d517bb61abc88762c9226cdfdfacb6ba56e76c742b WatchSource:0}: Error finding container 827ecbe6b3b684937f8897d517bb61abc88762c9226cdfdfacb6ba56e76c742b: Status 404 returned error can't find the container with id 827ecbe6b3b684937f8897d517bb61abc88762c9226cdfdfacb6ba56e76c742b Jul 29 00:59:01.488782 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:01.488738 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7d6d7bcf7c-62rxf"] Jul 29 00:59:01.491000 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:59:01.490975 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5dfce0d_d035_4da9_92bc_c6247d5b48c8.slice/crio-dc0e69fdcb2c20f568fcf58b1fdb2dcc1d93ddde708deadb9cbcac16f72068bd WatchSource:0}: Error finding container dc0e69fdcb2c20f568fcf58b1fdb2dcc1d93ddde708deadb9cbcac16f72068bd: Status 404 returned error can't find the container with id dc0e69fdcb2c20f568fcf58b1fdb2dcc1d93ddde708deadb9cbcac16f72068bd Jul 29 00:59:02.299380 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:02.299333 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" event={"ID":"e5dfce0d-d035-4da9-92bc-c6247d5b48c8","Type":"ContainerStarted","Data":"43fa015f8fb5828599a0a79c8f75a982b165eba36dfe686d856eaf0e0f54ecd3"} Jul 29 00:59:02.299573 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:02.299401 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" event={"ID":"e5dfce0d-d035-4da9-92bc-c6247d5b48c8","Type":"ContainerStarted","Data":"dc0e69fdcb2c20f568fcf58b1fdb2dcc1d93ddde708deadb9cbcac16f72068bd"} Jul 29 00:59:02.300412 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:02.300383 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx" event={"ID":"90ebb0ff-2b60-4c52-97cb-1cf589c21b61","Type":"ContainerStarted","Data":"827ecbe6b3b684937f8897d517bb61abc88762c9226cdfdfacb6ba56e76c742b"} Jul 29 00:59:02.318386 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:02.318335 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" podStartSLOduration=33.318317839 podStartE2EDuration="33.318317839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:59:02.316451788 +0000 UTC m=+292.587096096" watchObservedRunningTime="2026-07-29 00:59:02.318317839 +0000 UTC m=+292.588962147" Jul 29 00:59:02.344054 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:02.344025 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:59:02.346785 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:02.346760 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:59:03.307020 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:03.306932 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx" event={"ID":"90ebb0ff-2b60-4c52-97cb-1cf589c21b61","Type":"ContainerStarted","Data":"0a677775c720ed0a59775d453e3c2052fb125e59058dadad3ecbcedb5a685814"} Jul 29 00:59:03.307191 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:03.307167 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:59:03.308508 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:03.308487 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7d6d7bcf7c-62rxf" Jul 29 00:59:03.324725 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:03.324677 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-zt4lx" podStartSLOduration=32.772948981 podStartE2EDuration="34.32466249s" totalImagesPullingTime="1.551713509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:59:03.323217957 +0000 UTC m=+293.593862265" watchObservedRunningTime="2026-07-29 00:59:03.32466249 +0000 UTC m=+293.595306796" Jul 29 00:59:03.502649 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:03.502610 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bcsq5"] Jul 29 00:59:03.505767 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:03.505750 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bcsq5" Jul 29 00:59:03.507602 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:03.507580 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 29 00:59:03.507704 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:03.507627 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-c9lbk\"" Jul 29 00:59:03.512958 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:03.512899 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bcsq5"] Jul 29 00:59:03.639005 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:03.638970 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/60f6356c-85a2-462a-8855-df307d6093bc-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-bcsq5\" (UID: \"60f6356c-85a2-462a-8855-df307d6093bc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bcsq5" Jul 29 00:59:03.739523 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:03.739479 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/60f6356c-85a2-462a-8855-df307d6093bc-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-bcsq5\" (UID: \"60f6356c-85a2-462a-8855-df307d6093bc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bcsq5" Jul 29 00:59:03.741979 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:03.741945 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/60f6356c-85a2-462a-8855-df307d6093bc-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-bcsq5\" (UID: \"60f6356c-85a2-462a-8855-df307d6093bc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bcsq5" Jul 29 00:59:03.815974 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:03.815933 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bcsq5" Jul 29 00:59:03.946453 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:03.946423 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bcsq5"] Jul 29 00:59:03.949234 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:59:03.949207 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f6356c_85a2_462a_8855_df307d6093bc.slice/crio-5ea1cdadea9c6211d1602c12ff494c1a0d8b3e8d6b059a822866dd892bfa2378 WatchSource:0}: Error finding container 5ea1cdadea9c6211d1602c12ff494c1a0d8b3e8d6b059a822866dd892bfa2378: Status 404 returned error can't find the container with id 5ea1cdadea9c6211d1602c12ff494c1a0d8b3e8d6b059a822866dd892bfa2378 Jul 29 00:59:04.311609 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:04.311523 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bcsq5" event={"ID":"60f6356c-85a2-462a-8855-df307d6093bc","Type":"ContainerStarted","Data":"5ea1cdadea9c6211d1602c12ff494c1a0d8b3e8d6b059a822866dd892bfa2378"} Jul 29 00:59:05.316025 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:05.315941 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bcsq5" event={"ID":"60f6356c-85a2-462a-8855-df307d6093bc","Type":"ContainerStarted","Data":"c63aa13dc93031b3b12f6517f4c56e56c8a8488931dc940f4099c1316d13bf5e"} Jul 29 00:59:05.332200 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:05.332150 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bcsq5" podStartSLOduration=1.2640280879999999 podStartE2EDuration="2.332135316s" totalImagesPullingTime="1.068107228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:59:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:59:05.330843737 +0000 UTC m=+295.601488056" watchObservedRunningTime="2026-07-29 00:59:05.332135316 +0000 UTC m=+295.602779623" Jul 29 00:59:05.352572 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:05.352541 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e12d3020-b8bc-478a-979e-db748500760e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:59:05.354778 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:05.354757 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e12d3020-b8bc-478a-979e-db748500760e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ppxd8\" (UID: \"e12d3020-b8bc-478a-979e-db748500760e\") " pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:59:05.364745 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:05.364720 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ppxd8" Jul 29 00:59:05.488536 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:05.488511 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ppxd8"] Jul 29 00:59:05.491036 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:59:05.491006 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12d3020_b8bc_478a_979e_db748500760e.slice/crio-f843fe08b0ddb809c39aeb176d338ad751e324c1b9fa878866480c12581863b4 WatchSource:0}: Error finding container f843fe08b0ddb809c39aeb176d338ad751e324c1b9fa878866480c12581863b4: Status 404 returned error can't find the container with id f843fe08b0ddb809c39aeb176d338ad751e324c1b9fa878866480c12581863b4 Jul 29 00:59:06.322694 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:06.322656 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ppxd8" event={"ID":"e12d3020-b8bc-478a-979e-db748500760e","Type":"ContainerStarted","Data":"f843fe08b0ddb809c39aeb176d338ad751e324c1b9fa878866480c12581863b4"} Jul 29 00:59:06.323185 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:06.322726 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bcsq5" Jul 29 00:59:06.328497 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:06.328469 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-bcsq5" Jul 29 00:59:06.339213 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:59:06.339168 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-jx7tt" podUID="78bcb31a-2890-4527-b8f0-73cabeaecd3e" Jul 29 00:59:06.619917 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:06.619874 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-z2x8p"] Jul 29 00:59:06.638273 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:06.638244 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-z2x8p"] Jul 29 00:59:06.638441 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:06.638376 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-z2x8p" Jul 29 00:59:06.641234 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:06.641015 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 29 00:59:06.641234 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:06.641030 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 29 00:59:06.641234 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:06.641052 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 29 00:59:06.641234 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:06.641080 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-w7rv8\"" Jul 29 00:59:06.766377 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:06.766335 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbgl8\" (UniqueName: \"kubernetes.io/projected/d5fa44cc-c54b-48fe-bcd0-e27c4cb85318-kube-api-access-dbgl8\") pod \"prometheus-operator-84d69f794d-z2x8p\" (UID: \"d5fa44cc-c54b-48fe-bcd0-e27c4cb85318\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z2x8p" Jul 29 00:59:06.766568 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:06.766388 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5fa44cc-c54b-48fe-bcd0-e27c4cb85318-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-z2x8p\" (UID: \"d5fa44cc-c54b-48fe-bcd0-e27c4cb85318\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z2x8p" Jul 29 00:59:06.766568 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:06.766477 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5fa44cc-c54b-48fe-bcd0-e27c4cb85318-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-z2x8p\" (UID: \"d5fa44cc-c54b-48fe-bcd0-e27c4cb85318\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z2x8p" Jul 29 00:59:06.766568 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:06.766531 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5fa44cc-c54b-48fe-bcd0-e27c4cb85318-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-z2x8p\" (UID: \"d5fa44cc-c54b-48fe-bcd0-e27c4cb85318\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z2x8p" Jul 29 00:59:06.867723 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:06.867678 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbgl8\" (UniqueName: \"kubernetes.io/projected/d5fa44cc-c54b-48fe-bcd0-e27c4cb85318-kube-api-access-dbgl8\") pod \"prometheus-operator-84d69f794d-z2x8p\" (UID: \"d5fa44cc-c54b-48fe-bcd0-e27c4cb85318\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z2x8p" Jul 29 00:59:06.867962 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:06.867737 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5fa44cc-c54b-48fe-bcd0-e27c4cb85318-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-z2x8p\" (UID: \"d5fa44cc-c54b-48fe-bcd0-e27c4cb85318\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z2x8p" Jul 29 00:59:06.867962 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:06.867780 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5fa44cc-c54b-48fe-bcd0-e27c4cb85318-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-z2x8p\" (UID: \"d5fa44cc-c54b-48fe-bcd0-e27c4cb85318\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z2x8p" Jul 29 00:59:06.867962 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:06.867839 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5fa44cc-c54b-48fe-bcd0-e27c4cb85318-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-z2x8p\" (UID: \"d5fa44cc-c54b-48fe-bcd0-e27c4cb85318\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z2x8p" Jul 29 00:59:06.868190 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:59:06.867976 2543 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 29 00:59:06.868190 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:59:06.868059 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5fa44cc-c54b-48fe-bcd0-e27c4cb85318-prometheus-operator-tls podName:d5fa44cc-c54b-48fe-bcd0-e27c4cb85318 nodeName:}" failed. No retries permitted until 2026-07-29 00:59:07.368039706 +0000 UTC m=+297.638683994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/d5fa44cc-c54b-48fe-bcd0-e27c4cb85318-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-z2x8p" (UID: "d5fa44cc-c54b-48fe-bcd0-e27c4cb85318") : secret "prometheus-operator-tls" not found Jul 29 00:59:06.868562 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:06.868535 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5fa44cc-c54b-48fe-bcd0-e27c4cb85318-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-z2x8p\" (UID: \"d5fa44cc-c54b-48fe-bcd0-e27c4cb85318\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z2x8p" Jul 29 00:59:06.870576 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:06.870519 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5fa44cc-c54b-48fe-bcd0-e27c4cb85318-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-z2x8p\" (UID: \"d5fa44cc-c54b-48fe-bcd0-e27c4cb85318\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z2x8p" Jul 29 00:59:06.875561 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:06.875538 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbgl8\" (UniqueName: \"kubernetes.io/projected/d5fa44cc-c54b-48fe-bcd0-e27c4cb85318-kube-api-access-dbgl8\") pod \"prometheus-operator-84d69f794d-z2x8p\" (UID: \"d5fa44cc-c54b-48fe-bcd0-e27c4cb85318\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z2x8p" Jul 29 00:59:07.372051 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:07.372017 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5fa44cc-c54b-48fe-bcd0-e27c4cb85318-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-z2x8p\" (UID: \"d5fa44cc-c54b-48fe-bcd0-e27c4cb85318\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z2x8p" Jul 29 00:59:07.374774 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:07.374743 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d5fa44cc-c54b-48fe-bcd0-e27c4cb85318-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-z2x8p\" (UID: \"d5fa44cc-c54b-48fe-bcd0-e27c4cb85318\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-z2x8p" Jul 29 00:59:07.549587 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:07.549548 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-z2x8p" Jul 29 00:59:07.708787 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:07.708761 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-z2x8p"] Jul 29 00:59:07.712516 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:59:07.712490 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5fa44cc_c54b_48fe_bcd0_e27c4cb85318.slice/crio-7d684f648b61dba5c4e6ca5062632188d4fcd81f034367a7fb11899f04a9cb9e WatchSource:0}: Error finding container 7d684f648b61dba5c4e6ca5062632188d4fcd81f034367a7fb11899f04a9cb9e: Status 404 returned error can't find the container with id 7d684f648b61dba5c4e6ca5062632188d4fcd81f034367a7fb11899f04a9cb9e Jul 29 00:59:08.330897 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:08.330854 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-z2x8p" event={"ID":"d5fa44cc-c54b-48fe-bcd0-e27c4cb85318","Type":"ContainerStarted","Data":"7d684f648b61dba5c4e6ca5062632188d4fcd81f034367a7fb11899f04a9cb9e"} Jul 29 00:59:09.291429 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:09.291396 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7jhhc" Jul 29 00:59:10.505144 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:10.505118 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 00:59:10.818804 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:10.818735 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 00:59:10.831336 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:10.831312 2543 kubelet.go:1648] "Image garbage collection succeeded" Jul 29 00:59:11.337514 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:11.337481 2543 scope.go:117] "RemoveContainer" containerID="31af71b4261fc658a04b8eb2236d3bd3b21ce9daa5b46223cfcd2165ea8c66d6" Jul 29 00:59:11.337672 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:59:11.337651 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-dl64r_openshift-console-operator(c6ed4d34-e21f-445c-850b-2ee2e32aa6f0)\"" pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" podUID="c6ed4d34-e21f-445c-850b-2ee2e32aa6f0" Jul 29 00:59:11.343090 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:11.343062 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ppxd8" event={"ID":"e12d3020-b8bc-478a-979e-db748500760e","Type":"ContainerStarted","Data":"3ab79770ddab6d11e245e534416222abf43945bf92b674df6031ed273018bc6f"} Jul 29 00:59:11.344948 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:11.344926 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-z2x8p" event={"ID":"d5fa44cc-c54b-48fe-bcd0-e27c4cb85318","Type":"ContainerStarted","Data":"519cd15381399686e8e97b3d8b36291760acf9fb28a234b2b4751e4b6669121d"} Jul 29 00:59:11.345054 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:11.344963 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-z2x8p" event={"ID":"d5fa44cc-c54b-48fe-bcd0-e27c4cb85318","Type":"ContainerStarted","Data":"0c409961cff6356a936282db723332d9c7ef22fa30b03ce22c66aa061d3c5be9"} Jul 29 00:59:11.361294 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:11.361255 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-z2x8p" podStartSLOduration=2.235799302 podStartE2EDuration="5.361242674s" totalImagesPullingTime="3.125443372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:59:11.359757843 +0000 UTC m=+301.630402150" watchObservedRunningTime="2026-07-29 00:59:11.361242674 +0000 UTC m=+301.631887019" Jul 29 00:59:12.962936 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:12.962898 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w6wbc"] Jul 29 00:59:12.975732 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:12.975700 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:12.977847 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:12.977766 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f2tnk\"" Jul 29 00:59:12.977847 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:12.977767 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 29 00:59:12.978043 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:12.977868 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 29 00:59:12.978157 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:12.978134 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 29 00:59:13.126374 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.126331 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-node-exporter-tls\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.126549 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.126389 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-sys\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.126549 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.126415 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77hmt\" (UniqueName: \"kubernetes.io/projected/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-kube-api-access-77hmt\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.126549 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.126486 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.126661 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.126587 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-node-exporter-wtmp\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.126661 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.126641 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-node-exporter-textfile\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.126745 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.126708 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-root\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.126745 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.126732 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-metrics-client-ca\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.126923 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.126799 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-node-exporter-accelerators-collector-config\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.228224 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.228145 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-node-exporter-wtmp\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.228224 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.228194 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-node-exporter-textfile\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.228444 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.228247 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-root\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.228444 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.228274 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-metrics-client-ca\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.228444 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.228308 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-node-exporter-accelerators-collector-config\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.228444 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.228333 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-node-exporter-tls\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.228444 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.228345 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-node-exporter-wtmp\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.228444 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.228363 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-sys\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.228444 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.228355 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-root\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.228444 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.228384 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77hmt\" (UniqueName: \"kubernetes.io/projected/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-kube-api-access-77hmt\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.228444 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.228412 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.228444 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.228419 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-sys\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.228907 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.228599 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-node-exporter-textfile\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.229020 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.228996 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-metrics-client-ca\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.229243 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.229217 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-node-exporter-accelerators-collector-config\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.231075 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.231054 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.231213 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.231191 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-node-exporter-tls\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.235555 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.235531 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77hmt\" (UniqueName: \"kubernetes.io/projected/4ce1f418-f28e-4c13-9fe2-e02d0dec2a90-kube-api-access-77hmt\") pod \"node-exporter-w6wbc\" (UID: \"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90\") " pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:13.287688 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:13.287655 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w6wbc" Jul 29 00:59:14.335510 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:59:14.335478 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce1f418_f28e_4c13_9fe2_e02d0dec2a90.slice/crio-79baccb836644f6a3d8e188be69bc368d5983a01ad51aeb77e8d7fa0935e6d00 WatchSource:0}: Error finding container 79baccb836644f6a3d8e188be69bc368d5983a01ad51aeb77e8d7fa0935e6d00: Status 404 returned error can't find the container with id 79baccb836644f6a3d8e188be69bc368d5983a01ad51aeb77e8d7fa0935e6d00 Jul 29 00:59:14.337384 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:14.337368 2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 00:59:14.355525 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:14.355498 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w6wbc" event={"ID":"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90","Type":"ContainerStarted","Data":"79baccb836644f6a3d8e188be69bc368d5983a01ad51aeb77e8d7fa0935e6d00"} Jul 29 00:59:15.274363 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:15.274308 2543 patch_prober.go:28] interesting pod/image-registry-684c9f4678-nd9lv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 29 00:59:15.274671 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:15.274624 2543 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" podUID="3613abf5-ad33-4253-bf63-c8f7ccd99759" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 00:59:15.361963 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:15.361925 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ppxd8" event={"ID":"e12d3020-b8bc-478a-979e-db748500760e","Type":"ContainerStarted","Data":"056df5122fb0cfb423361ff4b76cb0dfb1510450107caaa6c309bc4d060c3976"} Jul 29 00:59:16.367639 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:16.367603 2543 generic.go:356] "Generic (PLEG): container finished" podID="4ce1f418-f28e-4c13-9fe2-e02d0dec2a90" containerID="fad998f41dfb9238d6ed2738722d4b34224293424cf63b0903fb4267faab9554" exitCode=0 Jul 29 00:59:16.368121 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:16.367681 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w6wbc" event={"ID":"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90","Type":"ContainerDied","Data":"fad998f41dfb9238d6ed2738722d4b34224293424cf63b0903fb4267faab9554"} Jul 29 00:59:17.281327 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:17.281296 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:59:18.376641 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:18.376603 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w6wbc" event={"ID":"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90","Type":"ContainerStarted","Data":"41a5322d3ff35cba2a91967f261de1416be1e035bb18ca43c467b4b19955a89f"} Jul 29 00:59:18.377159 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:18.376661 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w6wbc" event={"ID":"4ce1f418-f28e-4c13-9fe2-e02d0dec2a90","Type":"ContainerStarted","Data":"0484f0bfefb12a4d416328546fbf4bc8eb66c90fe62582ff0f2ad49a1602967e"} Jul 29 00:59:18.378405 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:18.378381 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ppxd8" event={"ID":"e12d3020-b8bc-478a-979e-db748500760e","Type":"ContainerStarted","Data":"0a7108e69be1f169dac5b24761d4d48757909d85bbdcdebdf2d5d51b04f84bfd"} Jul 29 00:59:18.398654 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:18.398607 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w6wbc" podStartSLOduration=5.259874563 podStartE2EDuration="6.398593898s" totalImagesPullingTime="1.064771036s" totalInitContainerRuntime="73.948299ms" isStatefulPod=true podCreationTimestamp="2026-07-29 00:59:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:59:18.398338366 +0000 UTC m=+308.668982673" watchObservedRunningTime="2026-07-29 00:59:18.398593898 +0000 UTC m=+308.669238205" Jul 29 00:59:18.415960 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:18.415911 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ppxd8" podStartSLOduration=33.147380315 podStartE2EDuration="45.415898006s" totalImagesPullingTime="12.268517691s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 00:58:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:59:18.414746613 +0000 UTC m=+308.685390948" watchObservedRunningTime="2026-07-29 00:59:18.415898006 +0000 UTC m=+308.686542312" Jul 29 00:59:20.032172 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:20.032138 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-684c9f4678-nd9lv"] Jul 29 00:59:21.337836 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:21.337775 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 00:59:24.338322 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:24.338290 2543 scope.go:117] "RemoveContainer" containerID="31af71b4261fc658a04b8eb2236d3bd3b21ce9daa5b46223cfcd2165ea8c66d6" Jul 29 00:59:24.404160 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:24.404139 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 00:59:25.103846 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:25.103791 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-wlvqj"] Jul 29 00:59:25.106967 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:25.106947 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-wlvqj" Jul 29 00:59:25.109131 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:25.109106 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-gbtr5\"" Jul 29 00:59:25.109131 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:25.109129 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 29 00:59:25.109295 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:25.109112 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 29 00:59:25.114878 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:25.114857 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-wlvqj"] Jul 29 00:59:25.228345 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:25.228309 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmz5w\" (UniqueName: \"kubernetes.io/projected/655d6892-5628-44aa-bb78-baa703d432f8-kube-api-access-qmz5w\") pod \"downloads-7c97dc4555-wlvqj\" (UID: \"655d6892-5628-44aa-bb78-baa703d432f8\") " pod="openshift-console/downloads-7c97dc4555-wlvqj" Jul 29 00:59:25.329701 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:25.329669 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmz5w\" (UniqueName: \"kubernetes.io/projected/655d6892-5628-44aa-bb78-baa703d432f8-kube-api-access-qmz5w\") pod \"downloads-7c97dc4555-wlvqj\" (UID: \"655d6892-5628-44aa-bb78-baa703d432f8\") " pod="openshift-console/downloads-7c97dc4555-wlvqj" Jul 29 00:59:25.338650 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:25.338629 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmz5w\" (UniqueName: \"kubernetes.io/projected/655d6892-5628-44aa-bb78-baa703d432f8-kube-api-access-qmz5w\") pod \"downloads-7c97dc4555-wlvqj\" (UID: \"655d6892-5628-44aa-bb78-baa703d432f8\") " pod="openshift-console/downloads-7c97dc4555-wlvqj" Jul 29 00:59:25.409425 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:25.409351 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 00:59:25.409559 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:25.409441 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" event={"ID":"c6ed4d34-e21f-445c-850b-2ee2e32aa6f0","Type":"ContainerStarted","Data":"462fc824dcb83e29b4a7021ba0fd8fedd682c498db08368d01c962c93d89662c"} Jul 29 00:59:25.409801 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:25.409781 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" Jul 29 00:59:25.414586 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:25.414565 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" Jul 29 00:59:25.415706 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:25.415691 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-wlvqj" Jul 29 00:59:25.424371 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:25.424313 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-dl64r" podStartSLOduration=53.445795319 podStartE2EDuration="56.424295496s" totalImagesPullingTime="2.978500177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:59:25.423636326 +0000 UTC m=+315.694280634" watchObservedRunningTime="2026-07-29 00:59:25.424295496 +0000 UTC m=+315.694939807" Jul 29 00:59:25.545326 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:25.545297 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-wlvqj"] Jul 29 00:59:25.548635 ip-10-0-132-136 kubenswrapper[2543]: W0729 00:59:25.548608 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655d6892_5628_44aa_bb78_baa703d432f8.slice/crio-323b746a582f918f07b92dd8605a640486789f5d0c90f46091ccc9a0e2942029 WatchSource:0}: Error finding container 323b746a582f918f07b92dd8605a640486789f5d0c90f46091ccc9a0e2942029: Status 404 returned error can't find the container with id 323b746a582f918f07b92dd8605a640486789f5d0c90f46091ccc9a0e2942029 Jul 29 00:59:26.415810 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:26.415746 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-wlvqj" event={"ID":"655d6892-5628-44aa-bb78-baa703d432f8","Type":"ContainerStarted","Data":"323b746a582f918f07b92dd8605a640486789f5d0c90f46091ccc9a0e2942029"} Jul 29 00:59:42.483019 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:42.482988 2543 generic.go:356] "Generic (PLEG): container finished" podID="842fabe6-40df-4c93-bba3-568afdaa041e" containerID="da916dceff0599eaadd8e10e7051ad41795d107ce025f3dde706c78fb98814b6" exitCode=0 Jul 29 00:59:42.483507 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:42.483063 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq" event={"ID":"842fabe6-40df-4c93-bba3-568afdaa041e","Type":"ContainerDied","Data":"da916dceff0599eaadd8e10e7051ad41795d107ce025f3dde706c78fb98814b6"} Jul 29 00:59:42.483507 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:42.483456 2543 scope.go:117] "RemoveContainer" containerID="da916dceff0599eaadd8e10e7051ad41795d107ce025f3dde706c78fb98814b6" Jul 29 00:59:43.489695 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:43.489655 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq" event={"ID":"842fabe6-40df-4c93-bba3-568afdaa041e","Type":"ContainerStarted","Data":"fb78be024320f588f3212ef9ba5882c6257d4f4c7b6f8a5555ec5d088a237524"} Jul 29 00:59:43.491440 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:43.491409 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-wlvqj" event={"ID":"655d6892-5628-44aa-bb78-baa703d432f8","Type":"ContainerStarted","Data":"22e7763ebec3e50eafb6e7c4f3d5b4005452e4b2066a4af1b06aaab068b16ef9"} Jul 29 00:59:43.491617 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:43.491595 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-wlvqj" Jul 29 00:59:43.503702 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:43.503679 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-wlvqj" Jul 29 00:59:43.519358 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:43.519304 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-wlvqj" podStartSLOduration=1.35252031 podStartE2EDuration="18.519284508s" totalImagesPullingTime="17.166764198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 00:59:43.518085028 +0000 UTC m=+333.788729337" watchObservedRunningTime="2026-07-29 00:59:43.519284508 +0000 UTC m=+333.789928815" Jul 29 00:59:45.051379 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.051302 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" podUID="3613abf5-ad33-4253-bf63-c8f7ccd99759" containerName="registry" containerID="cri-o://ed2e36729f73fa1d246e0e4e452737a15a617b21609e318d8a7063514890f274" gracePeriod=30 Jul 29 00:59:45.345122 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.345089 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:59:45.505141 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.505103 2543 generic.go:356] "Generic (PLEG): container finished" podID="3613abf5-ad33-4253-bf63-c8f7ccd99759" containerID="ed2e36729f73fa1d246e0e4e452737a15a617b21609e318d8a7063514890f274" exitCode=0 Jul 29 00:59:45.505331 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.505164 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" Jul 29 00:59:45.505331 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.505189 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" event={"ID":"3613abf5-ad33-4253-bf63-c8f7ccd99759","Type":"ContainerDied","Data":"ed2e36729f73fa1d246e0e4e452737a15a617b21609e318d8a7063514890f274"} Jul 29 00:59:45.505331 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.505243 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-684c9f4678-nd9lv" event={"ID":"3613abf5-ad33-4253-bf63-c8f7ccd99759","Type":"ContainerDied","Data":"03f19e1f59a558bb3abe75aa6a94a38bd2140fa5b9b288211443bdef19e80dc0"} Jul 29 00:59:45.505331 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.505273 2543 scope.go:117] "RemoveContainer" containerID="ed2e36729f73fa1d246e0e4e452737a15a617b21609e318d8a7063514890f274" Jul 29 00:59:45.514852 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.514800 2543 scope.go:117] "RemoveContainer" containerID="ed2e36729f73fa1d246e0e4e452737a15a617b21609e318d8a7063514890f274" Jul 29 00:59:45.515148 ip-10-0-132-136 kubenswrapper[2543]: E0729 00:59:45.515121 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed2e36729f73fa1d246e0e4e452737a15a617b21609e318d8a7063514890f274\": container with ID starting with ed2e36729f73fa1d246e0e4e452737a15a617b21609e318d8a7063514890f274 not found: ID does not exist" containerID="ed2e36729f73fa1d246e0e4e452737a15a617b21609e318d8a7063514890f274" Jul 29 00:59:45.515245 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.515158 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed2e36729f73fa1d246e0e4e452737a15a617b21609e318d8a7063514890f274"} err="failed to get container status \"ed2e36729f73fa1d246e0e4e452737a15a617b21609e318d8a7063514890f274\": rpc error: code = NotFound desc = could not find container \"ed2e36729f73fa1d246e0e4e452737a15a617b21609e318d8a7063514890f274\": container with ID starting with ed2e36729f73fa1d246e0e4e452737a15a617b21609e318d8a7063514890f274 not found: ID does not exist" Jul 29 00:59:45.516348 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.516324 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-certificates\") pod \"3613abf5-ad33-4253-bf63-c8f7ccd99759\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " Jul 29 00:59:45.516447 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.516355 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3613abf5-ad33-4253-bf63-c8f7ccd99759-ca-trust-extracted\") pod \"3613abf5-ad33-4253-bf63-c8f7ccd99759\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " Jul 29 00:59:45.516447 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.516382 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3613abf5-ad33-4253-bf63-c8f7ccd99759-installation-pull-secrets\") pod \"3613abf5-ad33-4253-bf63-c8f7ccd99759\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " Jul 29 00:59:45.516447 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.516412 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3613abf5-ad33-4253-bf63-c8f7ccd99759-trusted-ca\") pod \"3613abf5-ad33-4253-bf63-c8f7ccd99759\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " Jul 29 00:59:45.516595 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.516526 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-bound-sa-token\") pod \"3613abf5-ad33-4253-bf63-c8f7ccd99759\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " Jul 29 00:59:45.516595 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.516569 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3613abf5-ad33-4253-bf63-c8f7ccd99759-image-registry-private-configuration\") pod \"3613abf5-ad33-4253-bf63-c8f7ccd99759\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " Jul 29 00:59:45.516699 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.516650 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls\") pod \"3613abf5-ad33-4253-bf63-c8f7ccd99759\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " Jul 29 00:59:45.516753 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.516698 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqm2x\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-kube-api-access-rqm2x\") pod \"3613abf5-ad33-4253-bf63-c8f7ccd99759\" (UID: \"3613abf5-ad33-4253-bf63-c8f7ccd99759\") " Jul 29 00:59:45.516804 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.516746 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3613abf5-ad33-4253-bf63-c8f7ccd99759" (UID: "3613abf5-ad33-4253-bf63-c8f7ccd99759"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 00:59:45.516891 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.516832 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3613abf5-ad33-4253-bf63-c8f7ccd99759-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3613abf5-ad33-4253-bf63-c8f7ccd99759" (UID: "3613abf5-ad33-4253-bf63-c8f7ccd99759"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 00:59:45.517044 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.517018 2543 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-certificates\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 00:59:45.517044 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.517041 2543 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3613abf5-ad33-4253-bf63-c8f7ccd99759-trusted-ca\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 00:59:45.520144 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.519847 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3613abf5-ad33-4253-bf63-c8f7ccd99759" (UID: "3613abf5-ad33-4253-bf63-c8f7ccd99759"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 00:59:45.520144 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.519884 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3613abf5-ad33-4253-bf63-c8f7ccd99759" (UID: "3613abf5-ad33-4253-bf63-c8f7ccd99759"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 00:59:45.520144 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.520043 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3613abf5-ad33-4253-bf63-c8f7ccd99759-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3613abf5-ad33-4253-bf63-c8f7ccd99759" (UID: "3613abf5-ad33-4253-bf63-c8f7ccd99759"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 00:59:45.520341 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.520163 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3613abf5-ad33-4253-bf63-c8f7ccd99759-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3613abf5-ad33-4253-bf63-c8f7ccd99759" (UID: "3613abf5-ad33-4253-bf63-c8f7ccd99759"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 00:59:45.521451 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.521420 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-kube-api-access-rqm2x" (OuterVolumeSpecName: "kube-api-access-rqm2x") pod "3613abf5-ad33-4253-bf63-c8f7ccd99759" (UID: "3613abf5-ad33-4253-bf63-c8f7ccd99759"). InnerVolumeSpecName "kube-api-access-rqm2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 00:59:45.527912 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.527800 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3613abf5-ad33-4253-bf63-c8f7ccd99759-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3613abf5-ad33-4253-bf63-c8f7ccd99759" (UID: "3613abf5-ad33-4253-bf63-c8f7ccd99759"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 00:59:45.617873 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.617840 2543 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3613abf5-ad33-4253-bf63-c8f7ccd99759-installation-pull-secrets\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 00:59:45.617873 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.617874 2543 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-bound-sa-token\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 00:59:45.618100 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.617892 2543 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3613abf5-ad33-4253-bf63-c8f7ccd99759-image-registry-private-configuration\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 00:59:45.618100 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.617908 2543 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-registry-tls\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 00:59:45.618100 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.617921 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqm2x\" (UniqueName: \"kubernetes.io/projected/3613abf5-ad33-4253-bf63-c8f7ccd99759-kube-api-access-rqm2x\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 00:59:45.618100 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.617934 2543 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3613abf5-ad33-4253-bf63-c8f7ccd99759-ca-trust-extracted\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 00:59:45.827948 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.827915 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-684c9f4678-nd9lv"] Jul 29 00:59:45.830894 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:45.830864 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-684c9f4678-nd9lv"] Jul 29 00:59:46.343547 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:46.343516 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3613abf5-ad33-4253-bf63-c8f7ccd99759" path="/var/lib/kubelet/pods/3613abf5-ad33-4253-bf63-c8f7ccd99759/volumes" Jul 29 00:59:59.562828 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:59.562777 2543 generic.go:356] "Generic (PLEG): container finished" podID="20eff4d6-fdde-4bb9-a489-80daad0e2321" containerID="224550b704a18a7b19c6bfea72cd75594390027d2aab28527ef7afada6738ff8" exitCode=0 Jul 29 00:59:59.563297 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:59.562853 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-v9h56" event={"ID":"20eff4d6-fdde-4bb9-a489-80daad0e2321","Type":"ContainerDied","Data":"224550b704a18a7b19c6bfea72cd75594390027d2aab28527ef7afada6738ff8"} Jul 29 00:59:59.563297 ip-10-0-132-136 kubenswrapper[2543]: I0729 00:59:59.563157 2543 scope.go:117] "RemoveContainer" containerID="224550b704a18a7b19c6bfea72cd75594390027d2aab28527ef7afada6738ff8" Jul 29 01:00:00.568581 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:00:00.568548 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-v9h56" event={"ID":"20eff4d6-fdde-4bb9-a489-80daad0e2321","Type":"ContainerStarted","Data":"28a7bb3fa92f66f92fd42505e1daed8a49e50c33cb524e2d1c10ffac2095bb24"} Jul 29 01:00:23.120393 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:00:23.120354 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs\") pod \"network-metrics-daemon-jx7tt\" (UID: \"78bcb31a-2890-4527-b8f0-73cabeaecd3e\") " pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 01:00:23.122580 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:00:23.122559 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78bcb31a-2890-4527-b8f0-73cabeaecd3e-metrics-certs\") pod \"network-metrics-daemon-jx7tt\" (UID: \"78bcb31a-2890-4527-b8f0-73cabeaecd3e\") " pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 01:00:23.140459 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:00:23.140434 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bxdbf\"" Jul 29 01:00:23.149188 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:00:23.149167 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jx7tt" Jul 29 01:00:23.307503 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:00:23.307421 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jx7tt"] Jul 29 01:00:23.310140 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:00:23.310111 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78bcb31a_2890_4527_b8f0_73cabeaecd3e.slice/crio-bc6219b8baff96cc40ff8aa446bd67ef24c6c2a171368c55ddcbf027760a9a0b WatchSource:0}: Error finding container bc6219b8baff96cc40ff8aa446bd67ef24c6c2a171368c55ddcbf027760a9a0b: Status 404 returned error can't find the container with id bc6219b8baff96cc40ff8aa446bd67ef24c6c2a171368c55ddcbf027760a9a0b Jul 29 01:00:23.655938 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:00:23.655896 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jx7tt" event={"ID":"78bcb31a-2890-4527-b8f0-73cabeaecd3e","Type":"ContainerStarted","Data":"bc6219b8baff96cc40ff8aa446bd67ef24c6c2a171368c55ddcbf027760a9a0b"} Jul 29 01:00:25.664254 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:00:25.664216 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jx7tt" event={"ID":"78bcb31a-2890-4527-b8f0-73cabeaecd3e","Type":"ContainerStarted","Data":"438f11fce4151a93f0b44c186bcc3bf9ec85e553ff9c738ef0e3375b3f32a1f4"} Jul 29 01:00:25.664642 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:00:25.664268 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jx7tt" event={"ID":"78bcb31a-2890-4527-b8f0-73cabeaecd3e","Type":"ContainerStarted","Data":"f0007cdb2709cb1ef6298436501f0d22a5adf4070a0499a81e1e9ed323f2a5b5"} Jul 29 01:00:25.679473 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:00:25.679426 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jx7tt" podStartSLOduration=373.292412939 podStartE2EDuration="6m14.679412177s" totalImagesPullingTime="1.386999238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 00:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:00:25.678385814 +0000 UTC m=+375.949030121" watchObservedRunningTime="2026-07-29 01:00:25.679412177 +0000 UTC m=+375.950056484" Jul 29 01:01:40.622348 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:40.622312 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p"] Jul 29 01:01:40.622755 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:40.622584 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3613abf5-ad33-4253-bf63-c8f7ccd99759" containerName="registry" Jul 29 01:01:40.622755 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:40.622595 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="3613abf5-ad33-4253-bf63-c8f7ccd99759" containerName="registry" Jul 29 01:01:40.622755 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:40.622656 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="3613abf5-ad33-4253-bf63-c8f7ccd99759" containerName="registry" Jul 29 01:01:40.624577 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:40.624560 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p" Jul 29 01:01:40.626865 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:40.626842 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q6bqq\"" Jul 29 01:01:40.626865 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:40.626862 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 01:01:40.627013 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:40.626882 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 01:01:40.633993 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:40.633969 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p"] Jul 29 01:01:40.718312 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:40.718276 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb91133e-fc8d-4e74-9a2c-74be4769306c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p\" (UID: \"eb91133e-fc8d-4e74-9a2c-74be4769306c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p" Jul 29 01:01:40.718484 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:40.718333 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb91133e-fc8d-4e74-9a2c-74be4769306c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p\" (UID: \"eb91133e-fc8d-4e74-9a2c-74be4769306c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p" Jul 29 01:01:40.718484 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:40.718387 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm7km\" (UniqueName: \"kubernetes.io/projected/eb91133e-fc8d-4e74-9a2c-74be4769306c-kube-api-access-lm7km\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p\" (UID: \"eb91133e-fc8d-4e74-9a2c-74be4769306c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p" Jul 29 01:01:40.819614 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:40.819574 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb91133e-fc8d-4e74-9a2c-74be4769306c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p\" (UID: \"eb91133e-fc8d-4e74-9a2c-74be4769306c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p" Jul 29 01:01:40.819775 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:40.819635 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb91133e-fc8d-4e74-9a2c-74be4769306c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p\" (UID: \"eb91133e-fc8d-4e74-9a2c-74be4769306c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p" Jul 29 01:01:40.819775 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:40.819670 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm7km\" (UniqueName: \"kubernetes.io/projected/eb91133e-fc8d-4e74-9a2c-74be4769306c-kube-api-access-lm7km\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p\" (UID: \"eb91133e-fc8d-4e74-9a2c-74be4769306c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p" Jul 29 01:01:40.819981 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:40.819962 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb91133e-fc8d-4e74-9a2c-74be4769306c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p\" (UID: \"eb91133e-fc8d-4e74-9a2c-74be4769306c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p" Jul 29 01:01:40.820019 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:40.819993 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb91133e-fc8d-4e74-9a2c-74be4769306c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p\" (UID: \"eb91133e-fc8d-4e74-9a2c-74be4769306c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p" Jul 29 01:01:40.827376 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:40.827348 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm7km\" (UniqueName: \"kubernetes.io/projected/eb91133e-fc8d-4e74-9a2c-74be4769306c-kube-api-access-lm7km\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p\" (UID: \"eb91133e-fc8d-4e74-9a2c-74be4769306c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p" Jul 29 01:01:40.933706 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:40.933608 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p" Jul 29 01:01:41.054042 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:41.054010 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p"] Jul 29 01:01:41.057021 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:01:41.056991 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb91133e_fc8d_4e74_9a2c_74be4769306c.slice/crio-428f46c0c816b8f20e34378dfd9944bfba934d63509674c212ae27c581dae4e8 WatchSource:0}: Error finding container 428f46c0c816b8f20e34378dfd9944bfba934d63509674c212ae27c581dae4e8: Status 404 returned error can't find the container with id 428f46c0c816b8f20e34378dfd9944bfba934d63509674c212ae27c581dae4e8 Jul 29 01:01:41.939496 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:41.939454 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p" event={"ID":"eb91133e-fc8d-4e74-9a2c-74be4769306c","Type":"ContainerStarted","Data":"428f46c0c816b8f20e34378dfd9944bfba934d63509674c212ae27c581dae4e8"} Jul 29 01:01:49.972864 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:49.972807 2543 generic.go:356] "Generic (PLEG): container finished" podID="eb91133e-fc8d-4e74-9a2c-74be4769306c" containerID="9ffec9fd9cff26a66c60fbe99f1ae4a92ac63bf28abd973f8c777117ecd9fa62" exitCode=0 Jul 29 01:01:49.972864 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:49.972843 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p" event={"ID":"eb91133e-fc8d-4e74-9a2c-74be4769306c","Type":"ContainerDied","Data":"9ffec9fd9cff26a66c60fbe99f1ae4a92ac63bf28abd973f8c777117ecd9fa62"} Jul 29 01:01:53.988559 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:53.988520 2543 generic.go:356] "Generic (PLEG): container finished" podID="eb91133e-fc8d-4e74-9a2c-74be4769306c" containerID="42d8ba7fcc81e1e8d342417c4f9b217e48bf2e33031727b60f2ef7959abc98a2" exitCode=0 Jul 29 01:01:53.988953 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:01:53.988614 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p" event={"ID":"eb91133e-fc8d-4e74-9a2c-74be4769306c","Type":"ContainerDied","Data":"42d8ba7fcc81e1e8d342417c4f9b217e48bf2e33031727b60f2ef7959abc98a2"} Jul 29 01:02:01.021687 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:01.021602 2543 generic.go:356] "Generic (PLEG): container finished" podID="eb91133e-fc8d-4e74-9a2c-74be4769306c" containerID="f7214a5a7536b5caaa77126d495448a70a94ce61957a4874a10f277d896f4d9d" exitCode=0 Jul 29 01:02:01.022083 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:01.021682 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p" event={"ID":"eb91133e-fc8d-4e74-9a2c-74be4769306c","Type":"ContainerDied","Data":"f7214a5a7536b5caaa77126d495448a70a94ce61957a4874a10f277d896f4d9d"} Jul 29 01:02:02.146535 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:02.146511 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p" Jul 29 01:02:02.314526 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:02.314435 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm7km\" (UniqueName: \"kubernetes.io/projected/eb91133e-fc8d-4e74-9a2c-74be4769306c-kube-api-access-lm7km\") pod \"eb91133e-fc8d-4e74-9a2c-74be4769306c\" (UID: \"eb91133e-fc8d-4e74-9a2c-74be4769306c\") " Jul 29 01:02:02.314794 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:02.314535 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb91133e-fc8d-4e74-9a2c-74be4769306c-bundle\") pod \"eb91133e-fc8d-4e74-9a2c-74be4769306c\" (UID: \"eb91133e-fc8d-4e74-9a2c-74be4769306c\") " Jul 29 01:02:02.314794 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:02.314589 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb91133e-fc8d-4e74-9a2c-74be4769306c-util\") pod \"eb91133e-fc8d-4e74-9a2c-74be4769306c\" (UID: \"eb91133e-fc8d-4e74-9a2c-74be4769306c\") " Jul 29 01:02:02.315168 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:02.315143 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb91133e-fc8d-4e74-9a2c-74be4769306c-bundle" (OuterVolumeSpecName: "bundle") pod "eb91133e-fc8d-4e74-9a2c-74be4769306c" (UID: "eb91133e-fc8d-4e74-9a2c-74be4769306c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:02:02.316632 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:02.316608 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb91133e-fc8d-4e74-9a2c-74be4769306c-kube-api-access-lm7km" (OuterVolumeSpecName: "kube-api-access-lm7km") pod "eb91133e-fc8d-4e74-9a2c-74be4769306c" (UID: "eb91133e-fc8d-4e74-9a2c-74be4769306c"). InnerVolumeSpecName "kube-api-access-lm7km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:02:02.319412 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:02.319384 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb91133e-fc8d-4e74-9a2c-74be4769306c-util" (OuterVolumeSpecName: "util") pod "eb91133e-fc8d-4e74-9a2c-74be4769306c" (UID: "eb91133e-fc8d-4e74-9a2c-74be4769306c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:02:02.416079 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:02.416044 2543 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb91133e-fc8d-4e74-9a2c-74be4769306c-bundle\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:02:02.416079 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:02.416071 2543 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb91133e-fc8d-4e74-9a2c-74be4769306c-util\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:02:02.416079 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:02.416081 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm7km\" (UniqueName: \"kubernetes.io/projected/eb91133e-fc8d-4e74-9a2c-74be4769306c-kube-api-access-lm7km\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:02:03.031973 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:03.031941 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p" Jul 29 01:02:03.032153 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:03.031935 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s6q5p" event={"ID":"eb91133e-fc8d-4e74-9a2c-74be4769306c","Type":"ContainerDied","Data":"428f46c0c816b8f20e34378dfd9944bfba934d63509674c212ae27c581dae4e8"} Jul 29 01:02:03.032153 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:03.032064 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="428f46c0c816b8f20e34378dfd9944bfba934d63509674c212ae27c581dae4e8" Jul 29 01:02:14.229687 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:14.229650 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6zx8t"] Jul 29 01:02:14.230293 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:14.230043 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb91133e-fc8d-4e74-9a2c-74be4769306c" containerName="util" Jul 29 01:02:14.230293 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:14.230061 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb91133e-fc8d-4e74-9a2c-74be4769306c" containerName="util" Jul 29 01:02:14.230293 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:14.230092 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb91133e-fc8d-4e74-9a2c-74be4769306c" containerName="extract" Jul 29 01:02:14.230293 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:14.230102 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb91133e-fc8d-4e74-9a2c-74be4769306c" containerName="extract" Jul 29 01:02:14.230293 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:14.230118 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb91133e-fc8d-4e74-9a2c-74be4769306c" containerName="pull" Jul 29 01:02:14.230293 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:14.230127 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb91133e-fc8d-4e74-9a2c-74be4769306c" containerName="pull" Jul 29 01:02:14.230293 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:14.230195 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb91133e-fc8d-4e74-9a2c-74be4769306c" containerName="extract" Jul 29 01:02:14.232110 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:14.232090 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6zx8t" Jul 29 01:02:14.234440 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:14.234422 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 29 01:02:14.234934 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:14.234921 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 29 01:02:14.234934 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:14.234928 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-hfhvj\"" Jul 29 01:02:14.240743 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:14.240726 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6zx8t"] Jul 29 01:02:14.299696 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:14.299665 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whv8n\" (UniqueName: \"kubernetes.io/projected/a96bb174-c310-43f6-b700-f3e2cbc701c6-kube-api-access-whv8n\") pod \"cert-manager-webhook-6486ffd997-6zx8t\" (UID: \"a96bb174-c310-43f6-b700-f3e2cbc701c6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6zx8t" Jul 29 01:02:14.299873 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:14.299700 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a96bb174-c310-43f6-b700-f3e2cbc701c6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6zx8t\" (UID: \"a96bb174-c310-43f6-b700-f3e2cbc701c6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6zx8t" Jul 29 01:02:14.400695 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:14.400662 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whv8n\" (UniqueName: \"kubernetes.io/projected/a96bb174-c310-43f6-b700-f3e2cbc701c6-kube-api-access-whv8n\") pod \"cert-manager-webhook-6486ffd997-6zx8t\" (UID: \"a96bb174-c310-43f6-b700-f3e2cbc701c6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6zx8t" Jul 29 01:02:14.400862 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:14.400701 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a96bb174-c310-43f6-b700-f3e2cbc701c6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6zx8t\" (UID: \"a96bb174-c310-43f6-b700-f3e2cbc701c6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6zx8t" Jul 29 01:02:14.409728 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:14.409704 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a96bb174-c310-43f6-b700-f3e2cbc701c6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6zx8t\" (UID: \"a96bb174-c310-43f6-b700-f3e2cbc701c6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6zx8t" Jul 29 01:02:14.409975 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:14.409957 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whv8n\" (UniqueName: \"kubernetes.io/projected/a96bb174-c310-43f6-b700-f3e2cbc701c6-kube-api-access-whv8n\") pod \"cert-manager-webhook-6486ffd997-6zx8t\" (UID: \"a96bb174-c310-43f6-b700-f3e2cbc701c6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6zx8t" Jul 29 01:02:14.551526 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:14.551446 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6zx8t" Jul 29 01:02:14.716686 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:02:14.716658 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda96bb174_c310_43f6_b700_f3e2cbc701c6.slice/crio-95a95ea9c570548326d11d2ec90b4913def5d3180b5f02798afdbb37801dc8e6 WatchSource:0}: Error finding container 95a95ea9c570548326d11d2ec90b4913def5d3180b5f02798afdbb37801dc8e6: Status 404 returned error can't find the container with id 95a95ea9c570548326d11d2ec90b4913def5d3180b5f02798afdbb37801dc8e6 Jul 29 01:02:14.717909 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:14.717887 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6zx8t"] Jul 29 01:02:15.077537 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:15.077496 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6zx8t" event={"ID":"a96bb174-c310-43f6-b700-f3e2cbc701c6","Type":"ContainerStarted","Data":"95a95ea9c570548326d11d2ec90b4913def5d3180b5f02798afdbb37801dc8e6"} Jul 29 01:02:25.820686 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:25.820651 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gcl8c"] Jul 29 01:02:25.823006 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:25.822986 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gcl8c" Jul 29 01:02:25.825093 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:25.825070 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-2mpnp\"" Jul 29 01:02:25.830640 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:25.830595 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gcl8c"] Jul 29 01:02:25.884377 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:25.884330 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1ecdae5-3b6d-474d-89c8-2a8f445b5956-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gcl8c\" (UID: \"d1ecdae5-3b6d-474d-89c8-2a8f445b5956\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gcl8c" Jul 29 01:02:25.884521 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:25.884374 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmqvf\" (UniqueName: \"kubernetes.io/projected/d1ecdae5-3b6d-474d-89c8-2a8f445b5956-kube-api-access-pmqvf\") pod \"cert-manager-cainjector-559cdd8c87-gcl8c\" (UID: \"d1ecdae5-3b6d-474d-89c8-2a8f445b5956\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gcl8c" Jul 29 01:02:25.985246 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:25.985206 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1ecdae5-3b6d-474d-89c8-2a8f445b5956-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gcl8c\" (UID: \"d1ecdae5-3b6d-474d-89c8-2a8f445b5956\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gcl8c" Jul 29 01:02:25.985435 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:25.985256 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmqvf\" (UniqueName: \"kubernetes.io/projected/d1ecdae5-3b6d-474d-89c8-2a8f445b5956-kube-api-access-pmqvf\") pod \"cert-manager-cainjector-559cdd8c87-gcl8c\" (UID: \"d1ecdae5-3b6d-474d-89c8-2a8f445b5956\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gcl8c" Jul 29 01:02:25.992930 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:25.992901 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1ecdae5-3b6d-474d-89c8-2a8f445b5956-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gcl8c\" (UID: \"d1ecdae5-3b6d-474d-89c8-2a8f445b5956\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gcl8c" Jul 29 01:02:25.993042 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:25.992974 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmqvf\" (UniqueName: \"kubernetes.io/projected/d1ecdae5-3b6d-474d-89c8-2a8f445b5956-kube-api-access-pmqvf\") pod \"cert-manager-cainjector-559cdd8c87-gcl8c\" (UID: \"d1ecdae5-3b6d-474d-89c8-2a8f445b5956\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gcl8c" Jul 29 01:02:26.133941 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:26.133910 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gcl8c" Jul 29 01:02:26.274251 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:26.273548 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gcl8c"] Jul 29 01:02:27.124452 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:27.124408 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gcl8c" event={"ID":"d1ecdae5-3b6d-474d-89c8-2a8f445b5956","Type":"ContainerStarted","Data":"ba596fba225efcb1a54f12960fb5223678d05113bc2f9d33b86178e6b065531a"} Jul 29 01:02:28.129623 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:28.129587 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6zx8t" event={"ID":"a96bb174-c310-43f6-b700-f3e2cbc701c6","Type":"ContainerStarted","Data":"844e36fa13f56ee02ef5d185871add5fe15ed3de930d90c42dea8e3da6fdb089"} Jul 29 01:02:28.130023 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:28.129662 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6zx8t" Jul 29 01:02:28.145702 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:28.145652 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-6zx8t" podStartSLOduration=1.511573926 podStartE2EDuration="14.14563774s" totalImagesPullingTime="12.634063814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:02:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:02:28.144910636 +0000 UTC m=+498.415554943" watchObservedRunningTime="2026-07-29 01:02:28.14563774 +0000 UTC m=+498.416282046" Jul 29 01:02:29.135021 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:29.134984 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gcl8c" event={"ID":"d1ecdae5-3b6d-474d-89c8-2a8f445b5956","Type":"ContainerStarted","Data":"1043b0f023ecf87f3d754b16009c0c856803c232cf0f9afed405aaf0a01005db"} Jul 29 01:02:29.150955 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:29.150898 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gcl8c" podStartSLOduration=1.6844926120000001 podStartE2EDuration="4.150879137s" totalImagesPullingTime="2.466386525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:02:29.149277742 +0000 UTC m=+499.419922050" watchObservedRunningTime="2026-07-29 01:02:29.150879137 +0000 UTC m=+499.421523445" Jul 29 01:02:34.137634 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:34.137602 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6zx8t" Jul 29 01:02:46.367580 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:46.367550 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz"] Jul 29 01:02:46.375166 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:46.375140 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz" Jul 29 01:02:46.377903 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:46.377877 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 01:02:46.378668 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:46.378315 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-q6bqq\"" Jul 29 01:02:46.378668 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:46.378326 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 01:02:46.378668 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:46.378446 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz"] Jul 29 01:02:46.445909 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:46.445886 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfqlt\" (UniqueName: \"kubernetes.io/projected/25202fc5-bc7e-491b-8688-5a35b060a8c9-kube-api-access-hfqlt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz\" (UID: \"25202fc5-bc7e-491b-8688-5a35b060a8c9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz" Jul 29 01:02:46.446053 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:46.445918 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25202fc5-bc7e-491b-8688-5a35b060a8c9-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz\" (UID: \"25202fc5-bc7e-491b-8688-5a35b060a8c9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz" Jul 29 01:02:46.446053 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:46.446033 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25202fc5-bc7e-491b-8688-5a35b060a8c9-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz\" (UID: \"25202fc5-bc7e-491b-8688-5a35b060a8c9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz" Jul 29 01:02:46.547118 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:46.547087 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25202fc5-bc7e-491b-8688-5a35b060a8c9-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz\" (UID: \"25202fc5-bc7e-491b-8688-5a35b060a8c9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz" Jul 29 01:02:46.547289 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:46.547133 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfqlt\" (UniqueName: \"kubernetes.io/projected/25202fc5-bc7e-491b-8688-5a35b060a8c9-kube-api-access-hfqlt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz\" (UID: \"25202fc5-bc7e-491b-8688-5a35b060a8c9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz" Jul 29 01:02:46.547289 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:46.547154 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25202fc5-bc7e-491b-8688-5a35b060a8c9-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz\" (UID: \"25202fc5-bc7e-491b-8688-5a35b060a8c9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz" Jul 29 01:02:46.547470 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:46.547453 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25202fc5-bc7e-491b-8688-5a35b060a8c9-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz\" (UID: \"25202fc5-bc7e-491b-8688-5a35b060a8c9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz" Jul 29 01:02:46.547546 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:46.547520 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25202fc5-bc7e-491b-8688-5a35b060a8c9-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz\" (UID: \"25202fc5-bc7e-491b-8688-5a35b060a8c9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz" Jul 29 01:02:46.554513 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:46.554494 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfqlt\" (UniqueName: \"kubernetes.io/projected/25202fc5-bc7e-491b-8688-5a35b060a8c9-kube-api-access-hfqlt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz\" (UID: \"25202fc5-bc7e-491b-8688-5a35b060a8c9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz" Jul 29 01:02:46.685943 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:46.685861 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz" Jul 29 01:02:46.811245 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:46.811223 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz"] Jul 29 01:02:46.813255 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:02:46.813230 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25202fc5_bc7e_491b_8688_5a35b060a8c9.slice/crio-6f7a62380894d1a577e5831f403d1cea86c7c0f393b432b9554ad24f658a1e3f WatchSource:0}: Error finding container 6f7a62380894d1a577e5831f403d1cea86c7c0f393b432b9554ad24f658a1e3f: Status 404 returned error can't find the container with id 6f7a62380894d1a577e5831f403d1cea86c7c0f393b432b9554ad24f658a1e3f Jul 29 01:02:47.210589 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:47.210561 2543 generic.go:356] "Generic (PLEG): container finished" podID="25202fc5-bc7e-491b-8688-5a35b060a8c9" containerID="8aeb17fd34b96a2d8672d95818c5cf20ceb35b8efc1763b8821c4ceacf2eaf73" exitCode=0 Jul 29 01:02:47.210770 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:47.210600 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz" event={"ID":"25202fc5-bc7e-491b-8688-5a35b060a8c9","Type":"ContainerDied","Data":"8aeb17fd34b96a2d8672d95818c5cf20ceb35b8efc1763b8821c4ceacf2eaf73"} Jul 29 01:02:47.210770 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:47.210639 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz" event={"ID":"25202fc5-bc7e-491b-8688-5a35b060a8c9","Type":"ContainerStarted","Data":"6f7a62380894d1a577e5831f403d1cea86c7c0f393b432b9554ad24f658a1e3f"} Jul 29 01:02:48.216862 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:48.216761 2543 generic.go:356] "Generic (PLEG): container finished" podID="25202fc5-bc7e-491b-8688-5a35b060a8c9" containerID="e4950bc40c1e61cf274aeb12d4782ec1bd29232df28892f18f21d401eb300e3c" exitCode=0 Jul 29 01:02:48.216862 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:48.216847 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz" event={"ID":"25202fc5-bc7e-491b-8688-5a35b060a8c9","Type":"ContainerDied","Data":"e4950bc40c1e61cf274aeb12d4782ec1bd29232df28892f18f21d401eb300e3c"} Jul 29 01:02:49.225150 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:49.225119 2543 generic.go:356] "Generic (PLEG): container finished" podID="25202fc5-bc7e-491b-8688-5a35b060a8c9" containerID="8bc3b276b9068827b4fe16730c57a606c38804fac62721863b3c961373ac40f4" exitCode=0 Jul 29 01:02:49.225523 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:49.225210 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz" event={"ID":"25202fc5-bc7e-491b-8688-5a35b060a8c9","Type":"ContainerDied","Data":"8bc3b276b9068827b4fe16730c57a606c38804fac62721863b3c961373ac40f4"} Jul 29 01:02:50.356251 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:50.356225 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz" Jul 29 01:02:50.481272 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:50.481192 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25202fc5-bc7e-491b-8688-5a35b060a8c9-bundle\") pod \"25202fc5-bc7e-491b-8688-5a35b060a8c9\" (UID: \"25202fc5-bc7e-491b-8688-5a35b060a8c9\") " Jul 29 01:02:50.481272 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:50.481264 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25202fc5-bc7e-491b-8688-5a35b060a8c9-util\") pod \"25202fc5-bc7e-491b-8688-5a35b060a8c9\" (UID: \"25202fc5-bc7e-491b-8688-5a35b060a8c9\") " Jul 29 01:02:50.481477 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:50.481313 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfqlt\" (UniqueName: \"kubernetes.io/projected/25202fc5-bc7e-491b-8688-5a35b060a8c9-kube-api-access-hfqlt\") pod \"25202fc5-bc7e-491b-8688-5a35b060a8c9\" (UID: \"25202fc5-bc7e-491b-8688-5a35b060a8c9\") " Jul 29 01:02:50.481943 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:50.481916 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25202fc5-bc7e-491b-8688-5a35b060a8c9-bundle" (OuterVolumeSpecName: "bundle") pod "25202fc5-bc7e-491b-8688-5a35b060a8c9" (UID: "25202fc5-bc7e-491b-8688-5a35b060a8c9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:02:50.483342 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:50.483316 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25202fc5-bc7e-491b-8688-5a35b060a8c9-kube-api-access-hfqlt" (OuterVolumeSpecName: "kube-api-access-hfqlt") pod "25202fc5-bc7e-491b-8688-5a35b060a8c9" (UID: "25202fc5-bc7e-491b-8688-5a35b060a8c9"). InnerVolumeSpecName "kube-api-access-hfqlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:02:50.486928 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:50.486895 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25202fc5-bc7e-491b-8688-5a35b060a8c9-util" (OuterVolumeSpecName: "util") pod "25202fc5-bc7e-491b-8688-5a35b060a8c9" (UID: "25202fc5-bc7e-491b-8688-5a35b060a8c9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:02:50.582181 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:50.582150 2543 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/25202fc5-bc7e-491b-8688-5a35b060a8c9-bundle\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:02:50.582181 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:50.582174 2543 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/25202fc5-bc7e-491b-8688-5a35b060a8c9-util\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:02:50.582181 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:50.582185 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfqlt\" (UniqueName: \"kubernetes.io/projected/25202fc5-bc7e-491b-8688-5a35b060a8c9-kube-api-access-hfqlt\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:02:51.234155 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:51.234117 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz" event={"ID":"25202fc5-bc7e-491b-8688-5a35b060a8c9","Type":"ContainerDied","Data":"6f7a62380894d1a577e5831f403d1cea86c7c0f393b432b9554ad24f658a1e3f"} Jul 29 01:02:51.234155 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:51.234154 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8qnlz" Jul 29 01:02:51.234360 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:51.234200 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f7a62380894d1a577e5831f403d1cea86c7c0f393b432b9554ad24f658a1e3f" Jul 29 01:02:59.423504 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.423464 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c7bdf789-cgxcg"] Jul 29 01:02:59.424003 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.423905 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25202fc5-bc7e-491b-8688-5a35b060a8c9" containerName="pull" Jul 29 01:02:59.424003 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.423923 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="25202fc5-bc7e-491b-8688-5a35b060a8c9" containerName="pull" Jul 29 01:02:59.424003 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.423939 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25202fc5-bc7e-491b-8688-5a35b060a8c9" containerName="extract" Jul 29 01:02:59.424003 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.423947 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="25202fc5-bc7e-491b-8688-5a35b060a8c9" containerName="extract" Jul 29 01:02:59.424003 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.423980 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25202fc5-bc7e-491b-8688-5a35b060a8c9" containerName="util" Jul 29 01:02:59.424003 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.423987 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="25202fc5-bc7e-491b-8688-5a35b060a8c9" containerName="util" Jul 29 01:02:59.424287 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.424071 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="25202fc5-bc7e-491b-8688-5a35b060a8c9" containerName="extract" Jul 29 01:02:59.426186 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.426164 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c7bdf789-cgxcg" Jul 29 01:02:59.428191 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.428170 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 29 01:02:59.428337 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.428322 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 29 01:02:59.428975 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.428959 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 29 01:02:59.429056 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.428991 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 29 01:02:59.429241 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.429226 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 29 01:02:59.429908 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.429894 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rzm24\"" Jul 29 01:02:59.437331 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.437309 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c7bdf789-cgxcg"] Jul 29 01:02:59.554110 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.554067 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/78787cbb-a635-4cad-8b4c-a2e8c434eaae-manager-config\") pod \"lws-controller-manager-6c7bdf789-cgxcg\" (UID: \"78787cbb-a635-4cad-8b4c-a2e8c434eaae\") " pod="openshift-lws-operator/lws-controller-manager-6c7bdf789-cgxcg" Jul 29 01:02:59.554308 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.554138 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78787cbb-a635-4cad-8b4c-a2e8c434eaae-cert\") pod \"lws-controller-manager-6c7bdf789-cgxcg\" (UID: \"78787cbb-a635-4cad-8b4c-a2e8c434eaae\") " pod="openshift-lws-operator/lws-controller-manager-6c7bdf789-cgxcg" Jul 29 01:02:59.554308 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.554160 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/78787cbb-a635-4cad-8b4c-a2e8c434eaae-metrics-cert\") pod \"lws-controller-manager-6c7bdf789-cgxcg\" (UID: \"78787cbb-a635-4cad-8b4c-a2e8c434eaae\") " pod="openshift-lws-operator/lws-controller-manager-6c7bdf789-cgxcg" Jul 29 01:02:59.554308 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.554176 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx77s\" (UniqueName: \"kubernetes.io/projected/78787cbb-a635-4cad-8b4c-a2e8c434eaae-kube-api-access-jx77s\") pod \"lws-controller-manager-6c7bdf789-cgxcg\" (UID: \"78787cbb-a635-4cad-8b4c-a2e8c434eaae\") " pod="openshift-lws-operator/lws-controller-manager-6c7bdf789-cgxcg" Jul 29 01:02:59.655560 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.655522 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78787cbb-a635-4cad-8b4c-a2e8c434eaae-cert\") pod \"lws-controller-manager-6c7bdf789-cgxcg\" (UID: \"78787cbb-a635-4cad-8b4c-a2e8c434eaae\") " pod="openshift-lws-operator/lws-controller-manager-6c7bdf789-cgxcg" Jul 29 01:02:59.655560 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.655563 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/78787cbb-a635-4cad-8b4c-a2e8c434eaae-metrics-cert\") pod \"lws-controller-manager-6c7bdf789-cgxcg\" (UID: \"78787cbb-a635-4cad-8b4c-a2e8c434eaae\") " pod="openshift-lws-operator/lws-controller-manager-6c7bdf789-cgxcg" Jul 29 01:02:59.655830 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.655580 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx77s\" (UniqueName: \"kubernetes.io/projected/78787cbb-a635-4cad-8b4c-a2e8c434eaae-kube-api-access-jx77s\") pod \"lws-controller-manager-6c7bdf789-cgxcg\" (UID: \"78787cbb-a635-4cad-8b4c-a2e8c434eaae\") " pod="openshift-lws-operator/lws-controller-manager-6c7bdf789-cgxcg" Jul 29 01:02:59.655830 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.655625 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/78787cbb-a635-4cad-8b4c-a2e8c434eaae-manager-config\") pod \"lws-controller-manager-6c7bdf789-cgxcg\" (UID: \"78787cbb-a635-4cad-8b4c-a2e8c434eaae\") " pod="openshift-lws-operator/lws-controller-manager-6c7bdf789-cgxcg" Jul 29 01:02:59.656304 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.656280 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/78787cbb-a635-4cad-8b4c-a2e8c434eaae-manager-config\") pod \"lws-controller-manager-6c7bdf789-cgxcg\" (UID: \"78787cbb-a635-4cad-8b4c-a2e8c434eaae\") " pod="openshift-lws-operator/lws-controller-manager-6c7bdf789-cgxcg" Jul 29 01:02:59.658143 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.658119 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78787cbb-a635-4cad-8b4c-a2e8c434eaae-cert\") pod \"lws-controller-manager-6c7bdf789-cgxcg\" (UID: \"78787cbb-a635-4cad-8b4c-a2e8c434eaae\") " pod="openshift-lws-operator/lws-controller-manager-6c7bdf789-cgxcg" Jul 29 01:02:59.658228 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.658128 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/78787cbb-a635-4cad-8b4c-a2e8c434eaae-metrics-cert\") pod \"lws-controller-manager-6c7bdf789-cgxcg\" (UID: \"78787cbb-a635-4cad-8b4c-a2e8c434eaae\") " pod="openshift-lws-operator/lws-controller-manager-6c7bdf789-cgxcg" Jul 29 01:02:59.670962 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.670940 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx77s\" (UniqueName: \"kubernetes.io/projected/78787cbb-a635-4cad-8b4c-a2e8c434eaae-kube-api-access-jx77s\") pod \"lws-controller-manager-6c7bdf789-cgxcg\" (UID: \"78787cbb-a635-4cad-8b4c-a2e8c434eaae\") " pod="openshift-lws-operator/lws-controller-manager-6c7bdf789-cgxcg" Jul 29 01:02:59.735913 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.735830 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c7bdf789-cgxcg" Jul 29 01:02:59.885959 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:02:59.885925 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78787cbb_a635_4cad_8b4c_a2e8c434eaae.slice/crio-2a66139d7da157601a4aa2fbf0735a476d063762b15f029a97089445c0dab6d9 WatchSource:0}: Error finding container 2a66139d7da157601a4aa2fbf0735a476d063762b15f029a97089445c0dab6d9: Status 404 returned error can't find the container with id 2a66139d7da157601a4aa2fbf0735a476d063762b15f029a97089445c0dab6d9 Jul 29 01:02:59.886084 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:02:59.886042 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c7bdf789-cgxcg"] Jul 29 01:03:00.270767 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.270731 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c7bdf789-cgxcg" event={"ID":"78787cbb-a635-4cad-8b4c-a2e8c434eaae","Type":"ContainerStarted","Data":"2a66139d7da157601a4aa2fbf0735a476d063762b15f029a97089445c0dab6d9"} Jul 29 01:03:00.470604 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.470570 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl"] Jul 29 01:03:00.476364 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.476347 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" Jul 29 01:03:00.482005 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.481786 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 29 01:03:00.482005 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.481941 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-88xjl\"" Jul 29 01:03:00.482452 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.482431 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 29 01:03:00.482573 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.482494 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 29 01:03:00.486399 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.486377 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 29 01:03:00.492929 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.492908 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl"] Jul 29 01:03:00.663952 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.663915 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5dq7\" (UniqueName: \"kubernetes.io/projected/66b5b8b1-f61a-4ff2-8e1a-8f006dc19156-kube-api-access-b5dq7\") pod \"opendatahub-operator-controller-manager-795cff746b-c2pjl\" (UID: \"66b5b8b1-f61a-4ff2-8e1a-8f006dc19156\") " pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" Jul 29 01:03:00.664105 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.663977 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/66b5b8b1-f61a-4ff2-8e1a-8f006dc19156-manifests\") pod \"opendatahub-operator-controller-manager-795cff746b-c2pjl\" (UID: \"66b5b8b1-f61a-4ff2-8e1a-8f006dc19156\") " pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" Jul 29 01:03:00.664105 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.664035 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/66b5b8b1-f61a-4ff2-8e1a-8f006dc19156-apiservice-cert\") pod \"opendatahub-operator-controller-manager-795cff746b-c2pjl\" (UID: \"66b5b8b1-f61a-4ff2-8e1a-8f006dc19156\") " pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" Jul 29 01:03:00.664105 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.664092 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66b5b8b1-f61a-4ff2-8e1a-8f006dc19156-tmp\") pod \"opendatahub-operator-controller-manager-795cff746b-c2pjl\" (UID: \"66b5b8b1-f61a-4ff2-8e1a-8f006dc19156\") " pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" Jul 29 01:03:00.664221 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.664117 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66b5b8b1-f61a-4ff2-8e1a-8f006dc19156-webhook-cert\") pod \"opendatahub-operator-controller-manager-795cff746b-c2pjl\" (UID: \"66b5b8b1-f61a-4ff2-8e1a-8f006dc19156\") " pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" Jul 29 01:03:00.764766 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.764731 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66b5b8b1-f61a-4ff2-8e1a-8f006dc19156-tmp\") pod \"opendatahub-operator-controller-manager-795cff746b-c2pjl\" (UID: \"66b5b8b1-f61a-4ff2-8e1a-8f006dc19156\") " pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" Jul 29 01:03:00.764967 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.764770 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66b5b8b1-f61a-4ff2-8e1a-8f006dc19156-webhook-cert\") pod \"opendatahub-operator-controller-manager-795cff746b-c2pjl\" (UID: \"66b5b8b1-f61a-4ff2-8e1a-8f006dc19156\") " pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" Jul 29 01:03:00.764967 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.764843 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5dq7\" (UniqueName: \"kubernetes.io/projected/66b5b8b1-f61a-4ff2-8e1a-8f006dc19156-kube-api-access-b5dq7\") pod \"opendatahub-operator-controller-manager-795cff746b-c2pjl\" (UID: \"66b5b8b1-f61a-4ff2-8e1a-8f006dc19156\") " pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" Jul 29 01:03:00.765080 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.764985 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/66b5b8b1-f61a-4ff2-8e1a-8f006dc19156-manifests\") pod \"opendatahub-operator-controller-manager-795cff746b-c2pjl\" (UID: \"66b5b8b1-f61a-4ff2-8e1a-8f006dc19156\") " pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" Jul 29 01:03:00.765080 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.765059 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/66b5b8b1-f61a-4ff2-8e1a-8f006dc19156-apiservice-cert\") pod \"opendatahub-operator-controller-manager-795cff746b-c2pjl\" (UID: \"66b5b8b1-f61a-4ff2-8e1a-8f006dc19156\") " pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" Jul 29 01:03:00.765328 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.765295 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/66b5b8b1-f61a-4ff2-8e1a-8f006dc19156-manifests\") pod \"opendatahub-operator-controller-manager-795cff746b-c2pjl\" (UID: \"66b5b8b1-f61a-4ff2-8e1a-8f006dc19156\") " pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" Jul 29 01:03:00.767033 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.767014 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66b5b8b1-f61a-4ff2-8e1a-8f006dc19156-tmp\") pod \"opendatahub-operator-controller-manager-795cff746b-c2pjl\" (UID: \"66b5b8b1-f61a-4ff2-8e1a-8f006dc19156\") " pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" Jul 29 01:03:00.767269 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.767250 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/66b5b8b1-f61a-4ff2-8e1a-8f006dc19156-apiservice-cert\") pod \"opendatahub-operator-controller-manager-795cff746b-c2pjl\" (UID: \"66b5b8b1-f61a-4ff2-8e1a-8f006dc19156\") " pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" Jul 29 01:03:00.767379 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.767362 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/66b5b8b1-f61a-4ff2-8e1a-8f006dc19156-webhook-cert\") pod \"opendatahub-operator-controller-manager-795cff746b-c2pjl\" (UID: \"66b5b8b1-f61a-4ff2-8e1a-8f006dc19156\") " pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" Jul 29 01:03:00.772614 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.772596 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5dq7\" (UniqueName: \"kubernetes.io/projected/66b5b8b1-f61a-4ff2-8e1a-8f006dc19156-kube-api-access-b5dq7\") pod \"opendatahub-operator-controller-manager-795cff746b-c2pjl\" (UID: \"66b5b8b1-f61a-4ff2-8e1a-8f006dc19156\") " pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" Jul 29 01:03:00.790627 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.790603 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" Jul 29 01:03:00.962781 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:00.962650 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl"] Jul 29 01:03:00.966892 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:03:00.966857 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b5b8b1_f61a_4ff2_8e1a_8f006dc19156.slice/crio-43dbc090b05da8590595a08ca005cc06e42906487ef181cf1ad46733c8c629c6 WatchSource:0}: Error finding container 43dbc090b05da8590595a08ca005cc06e42906487ef181cf1ad46733c8c629c6: Status 404 returned error can't find the container with id 43dbc090b05da8590595a08ca005cc06e42906487ef181cf1ad46733c8c629c6 Jul 29 01:03:01.275962 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:01.275879 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" event={"ID":"66b5b8b1-f61a-4ff2-8e1a-8f006dc19156","Type":"ContainerStarted","Data":"43dbc090b05da8590595a08ca005cc06e42906487ef181cf1ad46733c8c629c6"} Jul 29 01:03:05.297164 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:05.297122 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" event={"ID":"66b5b8b1-f61a-4ff2-8e1a-8f006dc19156","Type":"ContainerStarted","Data":"291762c7101e8b5c792efe95826b167299e52c83d198715a09d608b1bdeeb471"} Jul 29 01:03:05.298748 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:05.298723 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c7bdf789-cgxcg" event={"ID":"78787cbb-a635-4cad-8b4c-a2e8c434eaae","Type":"ContainerStarted","Data":"29e3712dd89cfebb1a69078f79a048b15eac097bf2d75a810489b3a3f76a9b90"} Jul 29 01:03:05.298889 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:05.298860 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6c7bdf789-cgxcg" Jul 29 01:03:05.334171 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:05.334118 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6c7bdf789-cgxcg" podStartSLOduration=1.06754677 podStartE2EDuration="6.334101468s" totalImagesPullingTime="5.266554698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:03:05.333270943 +0000 UTC m=+535.603915250" watchObservedRunningTime="2026-07-29 01:03:05.334101468 +0000 UTC m=+535.604745775" Jul 29 01:03:07.309303 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:07.309266 2543 generic.go:356] "Generic (PLEG): container finished" podID="66b5b8b1-f61a-4ff2-8e1a-8f006dc19156" containerID="291762c7101e8b5c792efe95826b167299e52c83d198715a09d608b1bdeeb471" exitCode=0 Jul 29 01:03:07.309693 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:07.309326 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" event={"ID":"66b5b8b1-f61a-4ff2-8e1a-8f006dc19156","Type":"ContainerDied","Data":"291762c7101e8b5c792efe95826b167299e52c83d198715a09d608b1bdeeb471"} Jul 29 01:03:08.315159 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:08.315121 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" event={"ID":"66b5b8b1-f61a-4ff2-8e1a-8f006dc19156","Type":"ContainerStarted","Data":"e61f3ba47f4a446a3a6d0316d769f812e53e8cd74ac6d9b05e21dce975105154"} Jul 29 01:03:08.315580 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:08.315355 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" Jul 29 01:03:08.335542 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:08.335490 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" podStartSLOduration=2.756631962 podStartE2EDuration="8.335474518s" totalImagesPullingTime="4.549230849s" totalInitContainerRuntime="1.029611707s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:03:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:03:08.333391142 +0000 UTC m=+538.604035448" watchObservedRunningTime="2026-07-29 01:03:08.335474518 +0000 UTC m=+538.606118824" Jul 29 01:03:16.306210 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:16.306177 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6c7bdf789-cgxcg" Jul 29 01:03:19.321463 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:19.321432 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-795cff746b-c2pjl" Jul 29 01:03:29.448663 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.448627 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs"] Jul 29 01:03:29.458147 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.458125 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.463676 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.463655 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-9qfhh\"" Jul 29 01:03:29.463790 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.463662 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 01:03:29.468652 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.468627 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs"] Jul 29 01:03:29.477919 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.477896 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.478020 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.477930 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcvg7\" (UniqueName: \"kubernetes.io/projected/41fb5fb9-528f-46d0-812e-e01bcf10707f-kube-api-access-gcvg7\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.478020 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.477999 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.478116 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.478060 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.478116 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.478083 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.478183 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.478149 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.478183 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.478172 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.478260 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.478191 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fb5fb9-528f-46d0-812e-e01bcf10707f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.478260 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.478212 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.539296 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.539261 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn"] Jul 29 01:03:29.542421 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.542404 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.552701 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.552670 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn"] Jul 29 01:03:29.579011 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.578981 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d526833f-d86b-457b-bb62-730fc80ba854-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.579200 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.579023 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.579200 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.579053 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d526833f-d86b-457b-bb62-730fc80ba854-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.579200 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.579097 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d526833f-d86b-457b-bb62-730fc80ba854-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.579200 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.579135 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.579200 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.579160 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d526833f-d86b-457b-bb62-730fc80ba854-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.579473 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.579200 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nh9p\" (UniqueName: \"kubernetes.io/projected/d526833f-d86b-457b-bb62-730fc80ba854-kube-api-access-7nh9p\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.579473 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.579245 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.579473 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.579287 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d526833f-d86b-457b-bb62-730fc80ba854-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.579473 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.579354 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.579473 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.579375 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.579473 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.579393 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d526833f-d86b-457b-bb62-730fc80ba854-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.579473 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.579414 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d526833f-d86b-457b-bb62-730fc80ba854-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.579473 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.579433 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.579473 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.579440 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fb5fb9-528f-46d0-812e-e01bcf10707f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.579932 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.579481 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.579932 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.579510 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d526833f-d86b-457b-bb62-730fc80ba854-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.579932 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.579540 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.579932 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.579548 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.579932 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.579574 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcvg7\" (UniqueName: \"kubernetes.io/projected/41fb5fb9-528f-46d0-812e-e01bcf10707f-kube-api-access-gcvg7\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.579932 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.579735 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.579932 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.579789 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.580191 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.580157 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fb5fb9-528f-46d0-812e-e01bcf10707f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.581673 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.581648 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.581673 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.581633 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.591500 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.591474 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcvg7\" (UniqueName: \"kubernetes.io/projected/41fb5fb9-528f-46d0-812e-e01bcf10707f-kube-api-access-gcvg7\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.596190 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.596171 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bcth9zs\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.680052 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.680012 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d526833f-d86b-457b-bb62-730fc80ba854-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.680222 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.680065 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d526833f-d86b-457b-bb62-730fc80ba854-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.680222 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.680175 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nh9p\" (UniqueName: \"kubernetes.io/projected/d526833f-d86b-457b-bb62-730fc80ba854-kube-api-access-7nh9p\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.680222 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.680217 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d526833f-d86b-457b-bb62-730fc80ba854-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.680338 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.680248 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d526833f-d86b-457b-bb62-730fc80ba854-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.680338 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.680267 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d526833f-d86b-457b-bb62-730fc80ba854-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.680412 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.680373 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d526833f-d86b-457b-bb62-730fc80ba854-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.680460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.680439 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d526833f-d86b-457b-bb62-730fc80ba854-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.680527 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.680467 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d526833f-d86b-457b-bb62-730fc80ba854-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.680632 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.680598 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d526833f-d86b-457b-bb62-730fc80ba854-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.680721 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.680694 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d526833f-d86b-457b-bb62-730fc80ba854-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.680866 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.680735 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d526833f-d86b-457b-bb62-730fc80ba854-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.680866 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.680838 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d526833f-d86b-457b-bb62-730fc80ba854-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.681004 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.680888 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d526833f-d86b-457b-bb62-730fc80ba854-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.682558 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.682535 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d526833f-d86b-457b-bb62-730fc80ba854-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.682747 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.682731 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d526833f-d86b-457b-bb62-730fc80ba854-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.689120 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.689101 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d526833f-d86b-457b-bb62-730fc80ba854-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.689781 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.689757 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nh9p\" (UniqueName: \"kubernetes.io/projected/d526833f-d86b-457b-bb62-730fc80ba854-kube-api-access-7nh9p\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn\" (UID: \"d526833f-d86b-457b-bb62-730fc80ba854\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.769011 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.768924 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:29.852954 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.852899 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:29.934992 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:29.934879 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs"] Jul 29 01:03:29.938013 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:03:29.937987 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fb5fb9_528f_46d0_812e_e01bcf10707f.slice/crio-b5d0fb53c491a554d35a365a02394f650f164ea76762813fce33eb290e650638 WatchSource:0}: Error finding container b5d0fb53c491a554d35a365a02394f650f164ea76762813fce33eb290e650638: Status 404 returned error can't find the container with id b5d0fb53c491a554d35a365a02394f650f164ea76762813fce33eb290e650638 Jul 29 01:03:30.001773 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.001718 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn"] Jul 29 01:03:30.004024 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:03:30.003992 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd526833f_d86b_457b_bb62_730fc80ba854.slice/crio-ff39cf449e25d984a6d471bd4c0ce9f06bd65e308c990cce72e2e84e95b90e37 WatchSource:0}: Error finding container ff39cf449e25d984a6d471bd4c0ce9f06bd65e308c990cce72e2e84e95b90e37: Status 404 returned error can't find the container with id ff39cf449e25d984a6d471bd4c0ce9f06bd65e308c990cce72e2e84e95b90e37 Jul 29 01:03:30.238367 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.238329 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6"] Jul 29 01:03:30.242864 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.242847 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.245802 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.245780 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 29 01:03:30.245911 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.245787 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7ftf7\"" Jul 29 01:03:30.246380 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.246363 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 29 01:03:30.247512 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.247493 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 29 01:03:30.270265 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.270243 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6"] Jul 29 01:03:30.287743 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.287713 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ae36f8-1394-42d6-ab5e-855207cbdd69-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-hb7g6\" (UID: \"a1ae36f8-1394-42d6-ab5e-855207cbdd69\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.287886 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.287753 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a1ae36f8-1394-42d6-ab5e-855207cbdd69-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-hb7g6\" (UID: \"a1ae36f8-1394-42d6-ab5e-855207cbdd69\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.287886 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.287784 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rd7l\" (UniqueName: \"kubernetes.io/projected/a1ae36f8-1394-42d6-ab5e-855207cbdd69-kube-api-access-7rd7l\") pod \"istiod-openshift-gateway-5677ccb49b-hb7g6\" (UID: \"a1ae36f8-1394-42d6-ab5e-855207cbdd69\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.287886 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.287843 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ae36f8-1394-42d6-ab5e-855207cbdd69-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-hb7g6\" (UID: \"a1ae36f8-1394-42d6-ab5e-855207cbdd69\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.287886 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.287868 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a1ae36f8-1394-42d6-ab5e-855207cbdd69-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-hb7g6\" (UID: \"a1ae36f8-1394-42d6-ab5e-855207cbdd69\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.288035 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.287975 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a1ae36f8-1394-42d6-ab5e-855207cbdd69-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-hb7g6\" (UID: \"a1ae36f8-1394-42d6-ab5e-855207cbdd69\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.288035 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.288014 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a1ae36f8-1394-42d6-ab5e-855207cbdd69-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-hb7g6\" (UID: \"a1ae36f8-1394-42d6-ab5e-855207cbdd69\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.388668 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.388636 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a1ae36f8-1394-42d6-ab5e-855207cbdd69-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-hb7g6\" (UID: \"a1ae36f8-1394-42d6-ab5e-855207cbdd69\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.388888 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.388688 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a1ae36f8-1394-42d6-ab5e-855207cbdd69-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-hb7g6\" (UID: \"a1ae36f8-1394-42d6-ab5e-855207cbdd69\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.388888 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.388715 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ae36f8-1394-42d6-ab5e-855207cbdd69-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-hb7g6\" (UID: \"a1ae36f8-1394-42d6-ab5e-855207cbdd69\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.388888 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.388742 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a1ae36f8-1394-42d6-ab5e-855207cbdd69-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-hb7g6\" (UID: \"a1ae36f8-1394-42d6-ab5e-855207cbdd69\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.388888 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.388763 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rd7l\" (UniqueName: \"kubernetes.io/projected/a1ae36f8-1394-42d6-ab5e-855207cbdd69-kube-api-access-7rd7l\") pod \"istiod-openshift-gateway-5677ccb49b-hb7g6\" (UID: \"a1ae36f8-1394-42d6-ab5e-855207cbdd69\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.388888 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.388785 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ae36f8-1394-42d6-ab5e-855207cbdd69-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-hb7g6\" (UID: \"a1ae36f8-1394-42d6-ab5e-855207cbdd69\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.388888 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.388799 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a1ae36f8-1394-42d6-ab5e-855207cbdd69-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-hb7g6\" (UID: \"a1ae36f8-1394-42d6-ab5e-855207cbdd69\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.389904 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.389872 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a1ae36f8-1394-42d6-ab5e-855207cbdd69-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-hb7g6\" (UID: \"a1ae36f8-1394-42d6-ab5e-855207cbdd69\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.391142 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.391118 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a1ae36f8-1394-42d6-ab5e-855207cbdd69-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-hb7g6\" (UID: \"a1ae36f8-1394-42d6-ab5e-855207cbdd69\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.391268 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.391125 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ae36f8-1394-42d6-ab5e-855207cbdd69-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-hb7g6\" (UID: \"a1ae36f8-1394-42d6-ab5e-855207cbdd69\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.391268 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.391171 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a1ae36f8-1394-42d6-ab5e-855207cbdd69-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-hb7g6\" (UID: \"a1ae36f8-1394-42d6-ab5e-855207cbdd69\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.391769 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.391751 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a1ae36f8-1394-42d6-ab5e-855207cbdd69-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-hb7g6\" (UID: \"a1ae36f8-1394-42d6-ab5e-855207cbdd69\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.396586 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.396565 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ae36f8-1394-42d6-ab5e-855207cbdd69-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-hb7g6\" (UID: \"a1ae36f8-1394-42d6-ab5e-855207cbdd69\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.397289 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.397270 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rd7l\" (UniqueName: \"kubernetes.io/projected/a1ae36f8-1394-42d6-ab5e-855207cbdd69-kube-api-access-7rd7l\") pod \"istiod-openshift-gateway-5677ccb49b-hb7g6\" (UID: \"a1ae36f8-1394-42d6-ab5e-855207cbdd69\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.406694 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.406664 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" event={"ID":"41fb5fb9-528f-46d0-812e-e01bcf10707f","Type":"ContainerStarted","Data":"b5d0fb53c491a554d35a365a02394f650f164ea76762813fce33eb290e650638"} Jul 29 01:03:30.407794 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.407766 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" event={"ID":"d526833f-d86b-457b-bb62-730fc80ba854","Type":"ContainerStarted","Data":"ff39cf449e25d984a6d471bd4c0ce9f06bd65e308c990cce72e2e84e95b90e37"} Jul 29 01:03:30.552291 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.552207 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:30.700036 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:30.700012 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6"] Jul 29 01:03:30.701759 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:03:30.701732 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ae36f8_1394_42d6_ab5e_855207cbdd69.slice/crio-37f00bcdf87c6a2d463438f671dc45d4f215d3f942cb053c287a586223791c42 WatchSource:0}: Error finding container 37f00bcdf87c6a2d463438f671dc45d4f215d3f942cb053c287a586223791c42: Status 404 returned error can't find the container with id 37f00bcdf87c6a2d463438f671dc45d4f215d3f942cb053c287a586223791c42 Jul 29 01:03:31.413082 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:31.413048 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" event={"ID":"a1ae36f8-1394-42d6-ab5e-855207cbdd69","Type":"ContainerStarted","Data":"37f00bcdf87c6a2d463438f671dc45d4f215d3f942cb053c287a586223791c42"} Jul 29 01:03:34.740771 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:34.740713 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:03:34.741248 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:34.740803 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:03:34.741248 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:34.740855 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:03:34.741721 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:34.741689 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:03:34.741792 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:34.741775 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:03:34.741870 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:34.741847 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:03:34.809226 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:34.809200 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:03:34.809295 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:34.809257 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:03:35.435608 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:35.435576 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" event={"ID":"41fb5fb9-528f-46d0-812e-e01bcf10707f","Type":"ContainerStarted","Data":"e09e06900036768f88aeead900ea19688b59b193a345dcceb11c05862c71ce30"} Jul 29 01:03:35.437110 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:35.437082 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" event={"ID":"a1ae36f8-1394-42d6-ab5e-855207cbdd69","Type":"ContainerStarted","Data":"e0f4c7f6d7b26d9b20858932700e61c8c5711be7b0d805510b022de8ec6af46b"} Jul 29 01:03:35.437242 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:35.437126 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:35.438654 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:35.438630 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" event={"ID":"d526833f-d86b-457b-bb62-730fc80ba854","Type":"ContainerStarted","Data":"8b1402622ad1e00c4175ffef0077c9ec4779a8664663c8f6c1a514c672d2379b"} Jul 29 01:03:35.439142 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:35.439114 2543 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-hb7g6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 01:03:35.439244 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:35.439157 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" podUID="a1ae36f8-1394-42d6-ab5e-855207cbdd69" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 01:03:35.515776 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:35.515730 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" podStartSLOduration=1.715279019 podStartE2EDuration="6.515715598s" totalImagesPullingTime="4.800436579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:03:35.476099666 +0000 UTC m=+565.746743973" watchObservedRunningTime="2026-07-29 01:03:35.515715598 +0000 UTC m=+565.786359903" Jul 29 01:03:35.516105 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:35.516077 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" podStartSLOduration=1.410945276 podStartE2EDuration="5.51606728s" totalImagesPullingTime="4.105122004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:03:35.513794225 +0000 UTC m=+565.784438532" watchObservedRunningTime="2026-07-29 01:03:35.51606728 +0000 UTC m=+565.786711565" Jul 29 01:03:35.573073 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:35.573028 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" podStartSLOduration=1.837541024 podStartE2EDuration="6.573013498s" totalImagesPullingTime="4.735472474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:03:35.572292214 +0000 UTC m=+565.842936521" watchObservedRunningTime="2026-07-29 01:03:35.573013498 +0000 UTC m=+565.843657805" Jul 29 01:03:35.770140 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:35.770054 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:35.771173 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:35.771152 2543 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body= Jul 29 01:03:35.771273 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:35.771198 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" podUID="41fb5fb9-528f-46d0-812e-e01bcf10707f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" Jul 29 01:03:35.853999 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:35.853967 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:35.858961 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:35.858939 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:36.443077 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:36.443050 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:36.443441 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:36.443420 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hb7g6" Jul 29 01:03:36.443851 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:36.443810 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn" Jul 29 01:03:36.513919 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:36.513884 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs"] Jul 29 01:03:36.770199 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:36.770122 2543 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body= Jul 29 01:03:36.770199 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:36.770185 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" podUID="41fb5fb9-528f-46d0-812e-e01bcf10707f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" Jul 29 01:03:37.769901 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:37.769867 2543 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body= Jul 29 01:03:37.770061 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:37.769928 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" podUID="41fb5fb9-528f-46d0-812e-e01bcf10707f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" Jul 29 01:03:38.452867 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:38.452796 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" podUID="41fb5fb9-528f-46d0-812e-e01bcf10707f" containerName="istio-proxy" containerID="cri-o://e09e06900036768f88aeead900ea19688b59b193a345dcceb11c05862c71ce30" gracePeriod=30 Jul 29 01:03:43.695739 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.695718 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:43.805395 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.805311 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fb5fb9-528f-46d0-812e-e01bcf10707f-istiod-ca-cert\") pod \"41fb5fb9-528f-46d0-812e-e01bcf10707f\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " Jul 29 01:03:43.805395 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.805351 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-workload-certs\") pod \"41fb5fb9-528f-46d0-812e-e01bcf10707f\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " Jul 29 01:03:43.805395 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.805393 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-envoy\") pod \"41fb5fb9-528f-46d0-812e-e01bcf10707f\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " Jul 29 01:03:43.805668 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.805420 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-data\") pod \"41fb5fb9-528f-46d0-812e-e01bcf10707f\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " Jul 29 01:03:43.805668 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.805460 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-credential-socket\") pod \"41fb5fb9-528f-46d0-812e-e01bcf10707f\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " Jul 29 01:03:43.805668 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.805503 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-podinfo\") pod \"41fb5fb9-528f-46d0-812e-e01bcf10707f\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " Jul 29 01:03:43.805668 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.805536 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-token\") pod \"41fb5fb9-528f-46d0-812e-e01bcf10707f\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " Jul 29 01:03:43.805668 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.805582 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-workload-socket\") pod \"41fb5fb9-528f-46d0-812e-e01bcf10707f\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " Jul 29 01:03:43.805668 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.805619 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcvg7\" (UniqueName: \"kubernetes.io/projected/41fb5fb9-528f-46d0-812e-e01bcf10707f-kube-api-access-gcvg7\") pod \"41fb5fb9-528f-46d0-812e-e01bcf10707f\" (UID: \"41fb5fb9-528f-46d0-812e-e01bcf10707f\") " Jul 29 01:03:43.805964 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.805740 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "41fb5fb9-528f-46d0-812e-e01bcf10707f" (UID: "41fb5fb9-528f-46d0-812e-e01bcf10707f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:03:43.805964 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.805755 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-data" (OuterVolumeSpecName: "istio-data") pod "41fb5fb9-528f-46d0-812e-e01bcf10707f" (UID: "41fb5fb9-528f-46d0-812e-e01bcf10707f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:03:43.805964 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.805770 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "41fb5fb9-528f-46d0-812e-e01bcf10707f" (UID: "41fb5fb9-528f-46d0-812e-e01bcf10707f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:03:43.806093 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.805976 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-data\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:03:43.806093 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.805990 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "41fb5fb9-528f-46d0-812e-e01bcf10707f" (UID: "41fb5fb9-528f-46d0-812e-e01bcf10707f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:03:43.806093 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.805998 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-credential-socket\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:03:43.806093 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.806014 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-workload-certs\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:03:43.806264 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.806090 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41fb5fb9-528f-46d0-812e-e01bcf10707f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "41fb5fb9-528f-46d0-812e-e01bcf10707f" (UID: "41fb5fb9-528f-46d0-812e-e01bcf10707f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:03:43.807938 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.807916 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41fb5fb9-528f-46d0-812e-e01bcf10707f-kube-api-access-gcvg7" (OuterVolumeSpecName: "kube-api-access-gcvg7") pod "41fb5fb9-528f-46d0-812e-e01bcf10707f" (UID: "41fb5fb9-528f-46d0-812e-e01bcf10707f"). InnerVolumeSpecName "kube-api-access-gcvg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:03:43.808015 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.807989 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-token" (OuterVolumeSpecName: "istio-token") pod "41fb5fb9-528f-46d0-812e-e01bcf10707f" (UID: "41fb5fb9-528f-46d0-812e-e01bcf10707f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:03:43.808147 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.808125 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "41fb5fb9-528f-46d0-812e-e01bcf10707f" (UID: "41fb5fb9-528f-46d0-812e-e01bcf10707f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 01:03:43.808147 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.808134 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "41fb5fb9-528f-46d0-812e-e01bcf10707f" (UID: "41fb5fb9-528f-46d0-812e-e01bcf10707f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:03:43.906730 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.906695 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-podinfo\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:03:43.906730 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.906722 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-token\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:03:43.906730 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.906732 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-workload-socket\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:03:43.906978 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.906741 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcvg7\" (UniqueName: \"kubernetes.io/projected/41fb5fb9-528f-46d0-812e-e01bcf10707f-kube-api-access-gcvg7\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:03:43.906978 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.906751 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fb5fb9-528f-46d0-812e-e01bcf10707f-istiod-ca-cert\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:03:43.906978 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:43.906761 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fb5fb9-528f-46d0-812e-e01bcf10707f-istio-envoy\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:03:44.481061 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:44.481021 2543 generic.go:356] "Generic (PLEG): container finished" podID="41fb5fb9-528f-46d0-812e-e01bcf10707f" containerID="e09e06900036768f88aeead900ea19688b59b193a345dcceb11c05862c71ce30" exitCode=0 Jul 29 01:03:44.481252 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:44.481088 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" Jul 29 01:03:44.481252 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:44.481102 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" event={"ID":"41fb5fb9-528f-46d0-812e-e01bcf10707f","Type":"ContainerDied","Data":"e09e06900036768f88aeead900ea19688b59b193a345dcceb11c05862c71ce30"} Jul 29 01:03:44.481252 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:44.481180 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs" event={"ID":"41fb5fb9-528f-46d0-812e-e01bcf10707f","Type":"ContainerDied","Data":"b5d0fb53c491a554d35a365a02394f650f164ea76762813fce33eb290e650638"} Jul 29 01:03:44.481252 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:44.481203 2543 scope.go:117] "RemoveContainer" containerID="e09e06900036768f88aeead900ea19688b59b193a345dcceb11c05862c71ce30" Jul 29 01:03:44.489674 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:44.489658 2543 scope.go:117] "RemoveContainer" containerID="e09e06900036768f88aeead900ea19688b59b193a345dcceb11c05862c71ce30" Jul 29 01:03:44.489944 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:03:44.489925 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e09e06900036768f88aeead900ea19688b59b193a345dcceb11c05862c71ce30\": container with ID starting with e09e06900036768f88aeead900ea19688b59b193a345dcceb11c05862c71ce30 not found: ID does not exist" containerID="e09e06900036768f88aeead900ea19688b59b193a345dcceb11c05862c71ce30" Jul 29 01:03:44.490000 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:44.489956 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e09e06900036768f88aeead900ea19688b59b193a345dcceb11c05862c71ce30"} err="failed to get container status \"e09e06900036768f88aeead900ea19688b59b193a345dcceb11c05862c71ce30\": rpc error: code = NotFound desc = could not find container \"e09e06900036768f88aeead900ea19688b59b193a345dcceb11c05862c71ce30\": container with ID starting with e09e06900036768f88aeead900ea19688b59b193a345dcceb11c05862c71ce30 not found: ID does not exist" Jul 29 01:03:44.496919 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:44.496899 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs"] Jul 29 01:03:44.502339 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:44.502317 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bcth9zs"] Jul 29 01:03:46.342285 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:03:46.342252 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41fb5fb9-528f-46d0-812e-e01bcf10707f" path="/var/lib/kubelet/pods/41fb5fb9-528f-46d0-812e-e01bcf10707f/volumes" Jul 29 01:04:10.529700 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:04:10.529569 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 01:04:10.847149 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:04:10.847124 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 01:06:03.730906 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.730828 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2"] Jul 29 01:06:03.731365 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.731142 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41fb5fb9-528f-46d0-812e-e01bcf10707f" containerName="istio-proxy" Jul 29 01:06:03.731365 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.731152 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="41fb5fb9-528f-46d0-812e-e01bcf10707f" containerName="istio-proxy" Jul 29 01:06:03.731365 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.731213 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="41fb5fb9-528f-46d0-812e-e01bcf10707f" containerName="istio-proxy" Jul 29 01:06:03.734178 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.734130 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.736476 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.736454 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-7g7tl\"" Jul 29 01:06:03.746410 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.746385 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2"] Jul 29 01:06:03.850997 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.850962 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e366d26d-8475-439a-a70d-65c9a247b4a6-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.851164 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.851004 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e366d26d-8475-439a-a70d-65c9a247b4a6-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.851164 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.851027 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e366d26d-8475-439a-a70d-65c9a247b4a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.851164 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.851046 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e366d26d-8475-439a-a70d-65c9a247b4a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.851164 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.851088 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e366d26d-8475-439a-a70d-65c9a247b4a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.851164 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.851111 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e366d26d-8475-439a-a70d-65c9a247b4a6-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.851164 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.851157 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e366d26d-8475-439a-a70d-65c9a247b4a6-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.851365 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.851186 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfx6m\" (UniqueName: \"kubernetes.io/projected/e366d26d-8475-439a-a70d-65c9a247b4a6-kube-api-access-dfx6m\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.851365 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.851209 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e366d26d-8475-439a-a70d-65c9a247b4a6-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.951905 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.951864 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e366d26d-8475-439a-a70d-65c9a247b4a6-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.952073 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.951911 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfx6m\" (UniqueName: \"kubernetes.io/projected/e366d26d-8475-439a-a70d-65c9a247b4a6-kube-api-access-dfx6m\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.952073 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.951941 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e366d26d-8475-439a-a70d-65c9a247b4a6-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.952073 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.951975 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e366d26d-8475-439a-a70d-65c9a247b4a6-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.952073 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.951997 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e366d26d-8475-439a-a70d-65c9a247b4a6-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.952073 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.952018 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e366d26d-8475-439a-a70d-65c9a247b4a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.952073 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.952046 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e366d26d-8475-439a-a70d-65c9a247b4a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.952369 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.952084 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e366d26d-8475-439a-a70d-65c9a247b4a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.952369 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.952116 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e366d26d-8475-439a-a70d-65c9a247b4a6-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.952470 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.952400 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e366d26d-8475-439a-a70d-65c9a247b4a6-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.952470 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.952453 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e366d26d-8475-439a-a70d-65c9a247b4a6-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.952601 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.952582 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e366d26d-8475-439a-a70d-65c9a247b4a6-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.952648 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.952627 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e366d26d-8475-439a-a70d-65c9a247b4a6-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.952933 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.952909 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e366d26d-8475-439a-a70d-65c9a247b4a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.954593 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.954566 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e366d26d-8475-439a-a70d-65c9a247b4a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.954810 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.954793 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e366d26d-8475-439a-a70d-65c9a247b4a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.962712 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.962684 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e366d26d-8475-439a-a70d-65c9a247b4a6-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:03.962889 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:03.962869 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfx6m\" (UniqueName: \"kubernetes.io/projected/e366d26d-8475-439a-a70d-65c9a247b4a6-kube-api-access-dfx6m\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lkmf2\" (UID: \"e366d26d-8475-439a-a70d-65c9a247b4a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:04.044013 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:04.043931 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:04.179797 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:04.179767 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2"] Jul 29 01:06:04.181561 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:06:04.181537 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode366d26d_8475_439a_a70d_65c9a247b4a6.slice/crio-af155378d89f603b4fa6d77ec32c0e2be6fdd2dd7058cc447151f6d9183cf547 WatchSource:0}: Error finding container af155378d89f603b4fa6d77ec32c0e2be6fdd2dd7058cc447151f6d9183cf547: Status 404 returned error can't find the container with id af155378d89f603b4fa6d77ec32c0e2be6fdd2dd7058cc447151f6d9183cf547 Jul 29 01:06:04.183731 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:04.183697 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:06:04.183831 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:04.183761 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:06:04.183831 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:04.183788 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:06:05.109078 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:05.109041 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" event={"ID":"e366d26d-8475-439a-a70d-65c9a247b4a6","Type":"ContainerStarted","Data":"be5d5f0f8189cec6aa3017a9c02a0967fe6894e313b3e3791ec7df5d00ed2b78"} Jul 29 01:06:05.109486 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:05.109104 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" event={"ID":"e366d26d-8475-439a-a70d-65c9a247b4a6","Type":"ContainerStarted","Data":"af155378d89f603b4fa6d77ec32c0e2be6fdd2dd7058cc447151f6d9183cf547"} Jul 29 01:06:05.128090 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:05.128041 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" podStartSLOduration=2.12802583 podStartE2EDuration="2.12802583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:06:05.126312915 +0000 UTC m=+715.396957224" watchObservedRunningTime="2026-07-29 01:06:05.12802583 +0000 UTC m=+715.398670137" Jul 29 01:06:06.044840 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:06.044782 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:06.049954 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:06.049929 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:06.113860 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:06.113808 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:06:06.114746 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:06:06.114730 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:07:32.492427 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:07:32.492306 2543 generic.go:356] "Generic (PLEG): container finished" podID="e366d26d-8475-439a-a70d-65c9a247b4a6" containerID="be5d5f0f8189cec6aa3017a9c02a0967fe6894e313b3e3791ec7df5d00ed2b78" exitCode=0 Jul 29 01:07:32.492427 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:07:32.492381 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" event={"ID":"e366d26d-8475-439a-a70d-65c9a247b4a6","Type":"ContainerDied","Data":"be5d5f0f8189cec6aa3017a9c02a0967fe6894e313b3e3791ec7df5d00ed2b78"} Jul 29 01:07:32.492851 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:07:32.492717 2543 scope.go:117] "RemoveContainer" containerID="be5d5f0f8189cec6aa3017a9c02a0967fe6894e313b3e3791ec7df5d00ed2b78" Jul 29 01:07:32.493625 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:07:32.493601 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:07:32.493677 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:07:32.493666 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:07:32.493710 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:07:32.493698 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:07:33.502078 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:07:33.502032 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" event={"ID":"e366d26d-8475-439a-a70d-65c9a247b4a6","Type":"ContainerStarted","Data":"e84a4bea487f3c85e3481af5660efce992c36839d73a436c689085cc81f8f6e1"} Jul 29 01:07:34.044969 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:07:34.044936 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:07:34.046295 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:07:34.046270 2543 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lkmf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 29 01:07:34.046418 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:07:34.046324 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" podUID="e366d26d-8475-439a-a70d-65c9a247b4a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 29 01:07:35.044304 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:07:35.044270 2543 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lkmf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 29 01:07:35.044685 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:07:35.044322 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" podUID="e366d26d-8475-439a-a70d-65c9a247b4a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 29 01:07:36.045006 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:07:36.044968 2543 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lkmf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 29 01:07:36.045398 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:07:36.045023 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" podUID="e366d26d-8475-439a-a70d-65c9a247b4a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 29 01:07:36.113779 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:07:36.113747 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:07:37.044299 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:07:37.044262 2543 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lkmf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 29 01:07:37.044470 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:07:37.044316 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" podUID="e366d26d-8475-439a-a70d-65c9a247b4a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 29 01:07:38.048117 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:07:38.048083 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:07:38.048962 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:07:38.048947 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lkmf2" Jul 29 01:08:10.487001 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.486969 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh"] Jul 29 01:08:10.490161 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.490142 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:10.492278 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.492250 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 29 01:08:10.492935 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.492918 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kjskh\"" Jul 29 01:08:10.492935 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.492918 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 29 01:08:10.493051 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.492941 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 29 01:08:10.498413 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.498388 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh"] Jul 29 01:08:10.613517 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.613484 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9334c1ac-c7c7-4414-93d4-142926871eae-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5hfh\" (UID: \"9334c1ac-c7c7-4414-93d4-142926871eae\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:10.613669 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.613536 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9334c1ac-c7c7-4414-93d4-142926871eae-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5hfh\" (UID: \"9334c1ac-c7c7-4414-93d4-142926871eae\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:10.613669 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.613606 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9334c1ac-c7c7-4414-93d4-142926871eae-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5hfh\" (UID: \"9334c1ac-c7c7-4414-93d4-142926871eae\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:10.613669 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.613629 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9334c1ac-c7c7-4414-93d4-142926871eae-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5hfh\" (UID: \"9334c1ac-c7c7-4414-93d4-142926871eae\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:10.613669 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.613649 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9334c1ac-c7c7-4414-93d4-142926871eae-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5hfh\" (UID: \"9334c1ac-c7c7-4414-93d4-142926871eae\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:10.613851 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.613671 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89qjm\" (UniqueName: \"kubernetes.io/projected/9334c1ac-c7c7-4414-93d4-142926871eae-kube-api-access-89qjm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5hfh\" (UID: \"9334c1ac-c7c7-4414-93d4-142926871eae\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:10.714933 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.714895 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9334c1ac-c7c7-4414-93d4-142926871eae-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5hfh\" (UID: \"9334c1ac-c7c7-4414-93d4-142926871eae\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:10.715118 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.714945 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9334c1ac-c7c7-4414-93d4-142926871eae-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5hfh\" (UID: \"9334c1ac-c7c7-4414-93d4-142926871eae\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:10.715118 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.714962 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9334c1ac-c7c7-4414-93d4-142926871eae-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5hfh\" (UID: \"9334c1ac-c7c7-4414-93d4-142926871eae\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:10.715118 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.714985 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9334c1ac-c7c7-4414-93d4-142926871eae-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5hfh\" (UID: \"9334c1ac-c7c7-4414-93d4-142926871eae\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:10.715118 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.715012 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89qjm\" (UniqueName: \"kubernetes.io/projected/9334c1ac-c7c7-4414-93d4-142926871eae-kube-api-access-89qjm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5hfh\" (UID: \"9334c1ac-c7c7-4414-93d4-142926871eae\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:10.715118 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.715059 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9334c1ac-c7c7-4414-93d4-142926871eae-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5hfh\" (UID: \"9334c1ac-c7c7-4414-93d4-142926871eae\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:10.715432 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.715410 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9334c1ac-c7c7-4414-93d4-142926871eae-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5hfh\" (UID: \"9334c1ac-c7c7-4414-93d4-142926871eae\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:10.715515 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.715463 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9334c1ac-c7c7-4414-93d4-142926871eae-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5hfh\" (UID: \"9334c1ac-c7c7-4414-93d4-142926871eae\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:10.715515 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.715493 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9334c1ac-c7c7-4414-93d4-142926871eae-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5hfh\" (UID: \"9334c1ac-c7c7-4414-93d4-142926871eae\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:10.717241 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.717220 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9334c1ac-c7c7-4414-93d4-142926871eae-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5hfh\" (UID: \"9334c1ac-c7c7-4414-93d4-142926871eae\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:10.717518 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.717501 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9334c1ac-c7c7-4414-93d4-142926871eae-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5hfh\" (UID: \"9334c1ac-c7c7-4414-93d4-142926871eae\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:10.722259 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.722237 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89qjm\" (UniqueName: \"kubernetes.io/projected/9334c1ac-c7c7-4414-93d4-142926871eae-kube-api-access-89qjm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r5hfh\" (UID: \"9334c1ac-c7c7-4414-93d4-142926871eae\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:10.801371 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.801301 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:10.924526 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.924498 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh"] Jul 29 01:08:10.926271 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:08:10.926242 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9334c1ac_c7c7_4414_93d4_142926871eae.slice/crio-a76e46a8d7a8422bb52337f6e10c98691f13fcae4446d69bd8ef19542ab15355 WatchSource:0}: Error finding container a76e46a8d7a8422bb52337f6e10c98691f13fcae4446d69bd8ef19542ab15355: Status 404 returned error can't find the container with id a76e46a8d7a8422bb52337f6e10c98691f13fcae4446d69bd8ef19542ab15355 Jul 29 01:08:10.930402 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:10.928278 2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 01:08:11.687534 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:11.687492 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" event={"ID":"9334c1ac-c7c7-4414-93d4-142926871eae","Type":"ContainerStarted","Data":"a76e46a8d7a8422bb52337f6e10c98691f13fcae4446d69bd8ef19542ab15355"} Jul 29 01:08:12.693757 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:12.693721 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" event={"ID":"9334c1ac-c7c7-4414-93d4-142926871eae","Type":"ContainerStarted","Data":"cc298fbb730df47462cc791a6c0b1fd3ac43c78fd56ecaeccd619129614b72fd"} Jul 29 01:08:12.710790 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:12.710739 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" podStartSLOduration=1.7134581789999999 podStartE2EDuration="2.710721415s" totalImagesPullingTime="997.263236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:08:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:08:12.7091008 +0000 UTC m=+842.979745108" watchObservedRunningTime="2026-07-29 01:08:12.710721415 +0000 UTC m=+842.981365721" Jul 29 01:08:20.802392 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:20.802342 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:20.802804 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:20.802643 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:20.814744 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:20.814720 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:21.745870 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:21.745833 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r5hfh" Jul 29 01:08:31.887713 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:31.887672 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf"] Jul 29 01:08:31.891202 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:31.891181 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:31.893551 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:31.893533 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 29 01:08:31.901286 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:31.901266 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf"] Jul 29 01:08:31.992044 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:31.992007 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78537497-df70-464f-b54b-ffdeda1aa8c2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf\" (UID: \"78537497-df70-464f-b54b-ffdeda1aa8c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:31.992044 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:31.992048 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78537497-df70-464f-b54b-ffdeda1aa8c2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf\" (UID: \"78537497-df70-464f-b54b-ffdeda1aa8c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:31.992270 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:31.992073 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78537497-df70-464f-b54b-ffdeda1aa8c2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf\" (UID: \"78537497-df70-464f-b54b-ffdeda1aa8c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:31.992270 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:31.992149 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78537497-df70-464f-b54b-ffdeda1aa8c2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf\" (UID: \"78537497-df70-464f-b54b-ffdeda1aa8c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:31.992270 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:31.992184 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78537497-df70-464f-b54b-ffdeda1aa8c2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf\" (UID: \"78537497-df70-464f-b54b-ffdeda1aa8c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:31.992367 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:31.992273 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlw8x\" (UniqueName: \"kubernetes.io/projected/78537497-df70-464f-b54b-ffdeda1aa8c2-kube-api-access-vlw8x\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf\" (UID: \"78537497-df70-464f-b54b-ffdeda1aa8c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:32.093089 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:32.093053 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlw8x\" (UniqueName: \"kubernetes.io/projected/78537497-df70-464f-b54b-ffdeda1aa8c2-kube-api-access-vlw8x\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf\" (UID: \"78537497-df70-464f-b54b-ffdeda1aa8c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:32.093089 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:32.093095 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78537497-df70-464f-b54b-ffdeda1aa8c2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf\" (UID: \"78537497-df70-464f-b54b-ffdeda1aa8c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:32.093399 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:32.093117 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78537497-df70-464f-b54b-ffdeda1aa8c2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf\" (UID: \"78537497-df70-464f-b54b-ffdeda1aa8c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:32.093399 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:32.093140 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78537497-df70-464f-b54b-ffdeda1aa8c2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf\" (UID: \"78537497-df70-464f-b54b-ffdeda1aa8c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:32.093399 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:32.093173 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78537497-df70-464f-b54b-ffdeda1aa8c2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf\" (UID: \"78537497-df70-464f-b54b-ffdeda1aa8c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:32.093399 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:32.093194 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78537497-df70-464f-b54b-ffdeda1aa8c2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf\" (UID: \"78537497-df70-464f-b54b-ffdeda1aa8c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:32.093623 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:32.093535 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78537497-df70-464f-b54b-ffdeda1aa8c2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf\" (UID: \"78537497-df70-464f-b54b-ffdeda1aa8c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:32.093623 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:32.093562 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78537497-df70-464f-b54b-ffdeda1aa8c2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf\" (UID: \"78537497-df70-464f-b54b-ffdeda1aa8c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:32.093691 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:32.093629 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78537497-df70-464f-b54b-ffdeda1aa8c2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf\" (UID: \"78537497-df70-464f-b54b-ffdeda1aa8c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:32.095474 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:32.095444 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78537497-df70-464f-b54b-ffdeda1aa8c2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf\" (UID: \"78537497-df70-464f-b54b-ffdeda1aa8c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:32.095624 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:32.095607 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78537497-df70-464f-b54b-ffdeda1aa8c2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf\" (UID: \"78537497-df70-464f-b54b-ffdeda1aa8c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:32.100717 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:32.100692 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlw8x\" (UniqueName: \"kubernetes.io/projected/78537497-df70-464f-b54b-ffdeda1aa8c2-kube-api-access-vlw8x\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf\" (UID: \"78537497-df70-464f-b54b-ffdeda1aa8c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:32.201879 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:32.201766 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:32.327866 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:32.327839 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf"] Jul 29 01:08:32.330149 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:08:32.330102 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78537497_df70_464f_b54b_ffdeda1aa8c2.slice/crio-f655c9a32e46da5085733794b543f249b91d63b331502f710dc936113be59f6c WatchSource:0}: Error finding container f655c9a32e46da5085733794b543f249b91d63b331502f710dc936113be59f6c: Status 404 returned error can't find the container with id f655c9a32e46da5085733794b543f249b91d63b331502f710dc936113be59f6c Jul 29 01:08:32.786709 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:32.786611 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" event={"ID":"78537497-df70-464f-b54b-ffdeda1aa8c2","Type":"ContainerStarted","Data":"474c6288a3b190664095924af692526ed264a9516daae862dcf408d3de9acc51"} Jul 29 01:08:32.786709 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:32.786669 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" event={"ID":"78537497-df70-464f-b54b-ffdeda1aa8c2","Type":"ContainerStarted","Data":"f655c9a32e46da5085733794b543f249b91d63b331502f710dc936113be59f6c"} Jul 29 01:08:32.803442 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:32.803389 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" podStartSLOduration=1.605719771 podStartE2EDuration="1.803373588s" totalImagesPullingTime="197.653817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:08:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:08:32.801904372 +0000 UTC m=+863.072548680" watchObservedRunningTime="2026-07-29 01:08:32.803373588 +0000 UTC m=+863.074017895" Jul 29 01:08:42.202613 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:42.202573 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:42.203146 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:42.202641 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:42.215157 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:42.215133 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:08:42.843448 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:08:42.843417 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf" Jul 29 01:09:10.555934 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:09:10.555904 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 01:09:10.875018 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:09:10.874993 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 01:14:10.580940 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:14:10.580910 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 01:14:10.901946 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:14:10.901872 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 01:19:10.607440 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:19:10.607412 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 01:19:10.929786 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:19:10.929706 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 01:24:10.633471 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:24:10.633444 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 01:24:10.963592 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:24:10.963515 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 01:25:19.795597 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.795558 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq"] Jul 29 01:25:19.798867 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.798850 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.801485 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.801464 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-bb686069-openshift-default-dockercfg-5qn4q\"" Jul 29 01:25:19.816176 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.816151 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istiod-ca-cert\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.816272 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.816186 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-credential-socket\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.816272 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.816211 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-workload-socket\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.816272 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.816248 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ntzl\" (UniqueName: \"kubernetes.io/projected/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-kube-api-access-2ntzl\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.816372 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.816308 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-envoy\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.816372 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.816344 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-podinfo\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.816435 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.816378 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-data\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.816435 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.816406 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-workload-certs\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.816435 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.816424 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-token\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.821035 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.821013 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq"] Jul 29 01:25:19.917573 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.917534 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-podinfo\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.917573 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.917572 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-data\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.917853 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.917599 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-workload-certs\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.917853 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.917739 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-token\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.918051 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.918029 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-data\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.918392 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.918358 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istiod-ca-cert\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.918515 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.918417 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-credential-socket\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.918515 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.918471 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-workload-socket\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.918515 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.918497 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ntzl\" (UniqueName: \"kubernetes.io/projected/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-kube-api-access-2ntzl\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.918680 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.918416 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-workload-certs\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.918680 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.918574 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-envoy\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.918909 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.918888 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-credential-socket\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.919077 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.919055 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istiod-ca-cert\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.919379 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.919356 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-workload-socket\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.921199 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.921170 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-podinfo\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.921313 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.921200 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-envoy\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.926289 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.926254 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-token\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:19.926738 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:19.926714 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ntzl\" (UniqueName: \"kubernetes.io/projected/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-kube-api-access-2ntzl\") pod \"e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:20.109271 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:20.109224 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:20.265578 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:20.265550 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq"] Jul 29 01:25:20.267765 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:25:20.267738 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0095ba_60b3_4ee1_8fdb_a756bc9edc65.slice/crio-875f5f4fb2cfde2f3a8483c035a54edb1c42a0f1a02f7933e7a7cb9040db1331 WatchSource:0}: Error finding container 875f5f4fb2cfde2f3a8483c035a54edb1c42a0f1a02f7933e7a7cb9040db1331: Status 404 returned error can't find the container with id 875f5f4fb2cfde2f3a8483c035a54edb1c42a0f1a02f7933e7a7cb9040db1331 Jul 29 01:25:20.270319 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:20.270290 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:25:20.270383 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:20.270354 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:25:20.270420 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:20.270382 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:25:20.391705 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:20.391617 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" event={"ID":"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65","Type":"ContainerStarted","Data":"54feb74a79c4d199ff6b822eb863f2986d8e1789287c2e38fd7f1b953dbbc6e0"} Jul 29 01:25:20.391705 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:20.391670 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" event={"ID":"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65","Type":"ContainerStarted","Data":"875f5f4fb2cfde2f3a8483c035a54edb1c42a0f1a02f7933e7a7cb9040db1331"} Jul 29 01:25:20.430060 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:20.430002 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" podStartSLOduration=1.429985845 podStartE2EDuration="1.429985845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:25:20.42860786 +0000 UTC m=+1870.699252170" watchObservedRunningTime="2026-07-29 01:25:20.429985845 +0000 UTC m=+1870.700630152" Jul 29 01:25:21.109464 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:21.109427 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:21.114424 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:21.114398 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:21.396415 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:21.396330 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:21.397289 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:21.397273 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:37.438068 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:37.438020 2543 patch_prober.go:28] interesting pod/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 01:25:37.438585 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:37.438117 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" podUID="9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 01:25:38.476414 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:38.476381 2543 generic.go:356] "Generic (PLEG): container finished" podID="9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" containerID="54feb74a79c4d199ff6b822eb863f2986d8e1789287c2e38fd7f1b953dbbc6e0" exitCode=0 Jul 29 01:25:38.476414 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:38.476418 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" event={"ID":"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65","Type":"ContainerDied","Data":"54feb74a79c4d199ff6b822eb863f2986d8e1789287c2e38fd7f1b953dbbc6e0"} Jul 29 01:25:38.476874 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:38.476781 2543 scope.go:117] "RemoveContainer" containerID="54feb74a79c4d199ff6b822eb863f2986d8e1789287c2e38fd7f1b953dbbc6e0" Jul 29 01:25:38.477840 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:38.477796 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:25:38.477907 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:38.477876 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:25:38.477941 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:38.477912 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:25:39.483541 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:39.483505 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" event={"ID":"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65","Type":"ContainerStarted","Data":"2f56b2b956829321f0db5bdfc92578e388ee6c81e403cb17fca3b2be749352a6"} Jul 29 01:25:40.110802 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:40.110336 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:40.110802 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:40.110464 2543 patch_prober.go:28] interesting pod/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Jul 29 01:25:40.110802 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:40.110513 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" podUID="9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Jul 29 01:25:41.110433 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:41.110394 2543 patch_prober.go:28] interesting pod/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Jul 29 01:25:41.111027 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:41.110995 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" podUID="9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Jul 29 01:25:42.110174 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:42.110137 2543 patch_prober.go:28] interesting pod/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Jul 29 01:25:42.110479 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:42.110451 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" podUID="9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Jul 29 01:25:43.110031 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:43.109987 2543 patch_prober.go:28] interesting pod/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Jul 29 01:25:43.110239 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:43.110075 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" podUID="9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Jul 29 01:25:44.113361 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:44.113330 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:44.113750 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:44.113586 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:44.114395 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:44.114379 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:25:47.270759 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:47.270724 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb686069-bbfzz"] Jul 29 01:25:47.274053 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:47.274035 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb686069-bbfzz" Jul 29 01:25:47.276200 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:47.276183 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 01:25:47.276284 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:47.276227 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjvns\"" Jul 29 01:25:47.281217 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:47.281192 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb686069-bbfzz"] Jul 29 01:25:47.373932 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:47.373903 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e347bc-7439-4c63-89fe-8b524124a417-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bb686069-bbfzz\" (UID: \"70e347bc-7439-4c63-89fe-8b524124a417\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb686069-bbfzz" Jul 29 01:25:47.474892 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:47.474858 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e347bc-7439-4c63-89fe-8b524124a417-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bb686069-bbfzz\" (UID: \"70e347bc-7439-4c63-89fe-8b524124a417\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb686069-bbfzz" Jul 29 01:25:47.475496 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:47.475468 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e347bc-7439-4c63-89fe-8b524124a417-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bb686069-bbfzz\" (UID: \"70e347bc-7439-4c63-89fe-8b524124a417\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb686069-bbfzz" Jul 29 01:25:47.586165 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:47.586138 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb686069-bbfzz" Jul 29 01:25:47.711365 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:47.711333 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb686069-bbfzz"] Jul 29 01:25:47.714760 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:25:47.714723 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e347bc_7439_4c63_89fe_8b524124a417.slice/crio-237d8e9c72d55663e4fca89dbff69076179266add72946c2e4cdcc5b4d11d34a WatchSource:0}: Error finding container 237d8e9c72d55663e4fca89dbff69076179266add72946c2e4cdcc5b4d11d34a: Status 404 returned error can't find the container with id 237d8e9c72d55663e4fca89dbff69076179266add72946c2e4cdcc5b4d11d34a Jul 29 01:25:47.716700 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:47.716681 2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 01:25:48.530309 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:48.530271 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb686069-bbfzz" event={"ID":"70e347bc-7439-4c63-89fe-8b524124a417","Type":"ContainerStarted","Data":"237d8e9c72d55663e4fca89dbff69076179266add72946c2e4cdcc5b4d11d34a"} Jul 29 01:25:50.542519 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:50.542430 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb686069-bbfzz" event={"ID":"70e347bc-7439-4c63-89fe-8b524124a417","Type":"ContainerStarted","Data":"7b39cf3670101a0fbbf24adebb5f8a6aec772067cbe5afb82466cb738ebfac49"} Jul 29 01:25:50.558531 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:50.558484 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb686069-bbfzz" podStartSLOduration=1.165499118 podStartE2EDuration="3.55846828s" totalImagesPullingTime="2.392969162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:25:50.557680544 +0000 UTC m=+1900.828324854" watchObservedRunningTime="2026-07-29 01:25:50.55846828 +0000 UTC m=+1900.829112602" Jul 29 01:25:51.548392 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:51.548355 2543 generic.go:356] "Generic (PLEG): container finished" podID="70e347bc-7439-4c63-89fe-8b524124a417" containerID="7b39cf3670101a0fbbf24adebb5f8a6aec772067cbe5afb82466cb738ebfac49" exitCode=0 Jul 29 01:25:51.548839 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:51.548393 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb686069-bbfzz" event={"ID":"70e347bc-7439-4c63-89fe-8b524124a417","Type":"ContainerDied","Data":"7b39cf3670101a0fbbf24adebb5f8a6aec772067cbe5afb82466cb738ebfac49"} Jul 29 01:25:52.682241 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:52.682216 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb686069-bbfzz" Jul 29 01:25:52.823197 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:52.823106 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e347bc-7439-4c63-89fe-8b524124a417-maas-api-service-ca\") pod \"70e347bc-7439-4c63-89fe-8b524124a417\" (UID: \"70e347bc-7439-4c63-89fe-8b524124a417\") " Jul 29 01:25:52.823452 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:52.823430 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e347bc-7439-4c63-89fe-8b524124a417-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "70e347bc-7439-4c63-89fe-8b524124a417" (UID: "70e347bc-7439-4c63-89fe-8b524124a417"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:25:52.924067 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:52.924035 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e347bc-7439-4c63-89fe-8b524124a417-maas-api-service-ca\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:25:53.560655 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:53.560626 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb686069-bbfzz" Jul 29 01:25:53.560844 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:53.560653 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb686069-bbfzz" event={"ID":"70e347bc-7439-4c63-89fe-8b524124a417","Type":"ContainerDied","Data":"237d8e9c72d55663e4fca89dbff69076179266add72946c2e4cdcc5b4d11d34a"} Jul 29 01:25:53.560844 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:25:53.560698 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="237d8e9c72d55663e4fca89dbff69076179266add72946c2e4cdcc5b4d11d34a" Jul 29 01:26:02.882206 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:02.882169 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb686069-bbfzz"] Jul 29 01:26:02.885593 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:02.885558 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bb686069-bbfzz"] Jul 29 01:26:03.015580 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:03.015549 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq"] Jul 29 01:26:03.015779 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:03.015753 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" podUID="9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" containerName="istio-proxy" containerID="cri-o://2f56b2b956829321f0db5bdfc92578e388ee6c81e403cb17fca3b2be749352a6" gracePeriod=30 Jul 29 01:26:04.051000 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.050963 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm"] Jul 29 01:26:04.051377 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.051350 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70e347bc-7439-4c63-89fe-8b524124a417" containerName="revoke-keys" Jul 29 01:26:04.051377 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.051363 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="70e347bc-7439-4c63-89fe-8b524124a417" containerName="revoke-keys" Jul 29 01:26:04.051447 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.051418 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="70e347bc-7439-4c63-89fe-8b524124a417" containerName="revoke-keys" Jul 29 01:26:04.054441 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.054425 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.057902 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.057876 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-aaa0c308-gw-openshift-default-dockercfg-x4bgr\"" Jul 29 01:26:04.075040 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.075018 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm"] Jul 29 01:26:04.120389 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.120359 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-credential-socket\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.120561 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.120397 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-workload-certs\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.120561 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.120428 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztj7w\" (UniqueName: \"kubernetes.io/projected/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-kube-api-access-ztj7w\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.120561 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.120452 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-envoy\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.120561 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.120474 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istiod-ca-cert\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.120561 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.120504 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-podinfo\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.120806 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.120584 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-data\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.120806 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.120611 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-workload-socket\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.120806 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.120634 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-token\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.221681 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.221645 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-credential-socket\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.221681 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.221688 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-workload-certs\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.221958 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.221710 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztj7w\" (UniqueName: \"kubernetes.io/projected/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-kube-api-access-ztj7w\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.221958 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.221740 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-envoy\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.221958 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.221761 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istiod-ca-cert\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.221958 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.221796 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-podinfo\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.221958 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.221890 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-data\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.221958 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.221914 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-workload-socket\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.221958 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.221937 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-token\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.222310 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.222116 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-credential-socket\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.222310 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.222192 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-workload-certs\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.222411 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.222326 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-data\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.222411 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.222375 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-workload-socket\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.222749 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.222727 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istiod-ca-cert\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.224103 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.224075 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-envoy\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.224410 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.224386 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-podinfo\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.230618 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.230592 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztj7w\" (UniqueName: \"kubernetes.io/projected/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-kube-api-access-ztj7w\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.231614 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.231593 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-token\") pod \"e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.343247 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.343216 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70e347bc-7439-4c63-89fe-8b524124a417" path="/var/lib/kubelet/pods/70e347bc-7439-4c63-89fe-8b524124a417/volumes" Jul 29 01:26:04.364466 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.364437 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:04.509433 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.509401 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm"] Jul 29 01:26:04.511015 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:26:04.510982 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5bd9b4_3986_416d_bdef_0b3b3143bfe8.slice/crio-82d27beb8dfa4769a8809f3570e3180368456b4f4b2cd1bcd78280d327f0a283 WatchSource:0}: Error finding container 82d27beb8dfa4769a8809f3570e3180368456b4f4b2cd1bcd78280d327f0a283: Status 404 returned error can't find the container with id 82d27beb8dfa4769a8809f3570e3180368456b4f4b2cd1bcd78280d327f0a283 Jul 29 01:26:04.513150 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.513118 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:26:04.513236 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.513181 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:26:04.513236 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.513208 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:26:04.616278 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.616195 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" event={"ID":"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8","Type":"ContainerStarted","Data":"758ec61d92183c7bebcc9c9e0bb1d0a41684512c3d095a3bd33838b8180b10f9"} Jul 29 01:26:04.616278 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.616249 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" event={"ID":"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8","Type":"ContainerStarted","Data":"82d27beb8dfa4769a8809f3570e3180368456b4f4b2cd1bcd78280d327f0a283"} Jul 29 01:26:04.639182 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:04.639130 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" podStartSLOduration=0.639114963 podStartE2EDuration="639.114963ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:26:04.637233634 +0000 UTC m=+1914.907877961" watchObservedRunningTime="2026-07-29 01:26:04.639114963 +0000 UTC m=+1914.909759269" Jul 29 01:26:05.365597 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:05.365564 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:05.370792 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:05.370763 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:05.621543 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:05.621459 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:05.622457 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:05.622440 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:08.275367 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.275307 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:26:08.355119 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.355087 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-workload-socket\") pod \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " Jul 29 01:26:08.355119 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.355119 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-token\") pod \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " Jul 29 01:26:08.355332 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.355141 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istiod-ca-cert\") pod \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " Jul 29 01:26:08.355332 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.355158 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-credential-socket\") pod \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " Jul 29 01:26:08.355332 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.355187 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ntzl\" (UniqueName: \"kubernetes.io/projected/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-kube-api-access-2ntzl\") pod \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " Jul 29 01:26:08.355332 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.355233 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-workload-certs\") pod \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " Jul 29 01:26:08.355332 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.355263 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-podinfo\") pod \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " Jul 29 01:26:08.355332 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.355319 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-envoy\") pod \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " Jul 29 01:26:08.355631 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.355340 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-data\") pod \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\" (UID: \"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65\") " Jul 29 01:26:08.355631 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.355416 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" (UID: "9c0095ba-60b3-4ee1-8fdb-a756bc9edc65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:26:08.355631 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.355487 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" (UID: "9c0095ba-60b3-4ee1-8fdb-a756bc9edc65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:26:08.355794 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.355672 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" (UID: "9c0095ba-60b3-4ee1-8fdb-a756bc9edc65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:26:08.355794 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.355694 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-workload-socket\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:26:08.355794 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.355715 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-credential-socket\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:26:08.355985 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.355787 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" (UID: "9c0095ba-60b3-4ee1-8fdb-a756bc9edc65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:26:08.356316 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.356291 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-data" (OuterVolumeSpecName: "istio-data") pod "9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" (UID: "9c0095ba-60b3-4ee1-8fdb-a756bc9edc65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:26:08.357450 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.357422 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-kube-api-access-2ntzl" (OuterVolumeSpecName: "kube-api-access-2ntzl") pod "9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" (UID: "9c0095ba-60b3-4ee1-8fdb-a756bc9edc65"). InnerVolumeSpecName "kube-api-access-2ntzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:26:08.357450 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.357422 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" (UID: "9c0095ba-60b3-4ee1-8fdb-a756bc9edc65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 01:26:08.357583 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.357452 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-token" (OuterVolumeSpecName: "istio-token") pod "9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" (UID: "9c0095ba-60b3-4ee1-8fdb-a756bc9edc65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:26:08.357583 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.357532 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" (UID: "9c0095ba-60b3-4ee1-8fdb-a756bc9edc65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:26:08.456160 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.456130 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-token\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:26:08.456160 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.456155 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istiod-ca-cert\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:26:08.456160 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.456166 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ntzl\" (UniqueName: \"kubernetes.io/projected/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-kube-api-access-2ntzl\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:26:08.456436 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.456177 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-workload-certs\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:26:08.456436 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.456186 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-podinfo\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:26:08.456436 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.456194 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-envoy\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:26:08.456436 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.456202 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65-istio-data\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:26:08.642658 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.642595 2543 generic.go:356] "Generic (PLEG): container finished" podID="9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" containerID="2f56b2b956829321f0db5bdfc92578e388ee6c81e403cb17fca3b2be749352a6" exitCode=0 Jul 29 01:26:08.642658 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.642632 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" event={"ID":"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65","Type":"ContainerDied","Data":"2f56b2b956829321f0db5bdfc92578e388ee6c81e403cb17fca3b2be749352a6"} Jul 29 01:26:08.642947 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.642702 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" event={"ID":"9c0095ba-60b3-4ee1-8fdb-a756bc9edc65","Type":"ContainerDied","Data":"875f5f4fb2cfde2f3a8483c035a54edb1c42a0f1a02f7933e7a7cb9040db1331"} Jul 29 01:26:08.642947 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.642732 2543 scope.go:117] "RemoveContainer" containerID="2f56b2b956829321f0db5bdfc92578e388ee6c81e403cb17fca3b2be749352a6" Jul 29 01:26:08.642947 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.642642 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq" Jul 29 01:26:08.652193 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.652173 2543 scope.go:117] "RemoveContainer" containerID="54feb74a79c4d199ff6b822eb863f2986d8e1789287c2e38fd7f1b953dbbc6e0" Jul 29 01:26:08.659778 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.659758 2543 scope.go:117] "RemoveContainer" containerID="2f56b2b956829321f0db5bdfc92578e388ee6c81e403cb17fca3b2be749352a6" Jul 29 01:26:08.660040 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:26:08.660019 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f56b2b956829321f0db5bdfc92578e388ee6c81e403cb17fca3b2be749352a6\": container with ID starting with 2f56b2b956829321f0db5bdfc92578e388ee6c81e403cb17fca3b2be749352a6 not found: ID does not exist" containerID="2f56b2b956829321f0db5bdfc92578e388ee6c81e403cb17fca3b2be749352a6" Jul 29 01:26:08.660116 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.660053 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f56b2b956829321f0db5bdfc92578e388ee6c81e403cb17fca3b2be749352a6"} err="failed to get container status \"2f56b2b956829321f0db5bdfc92578e388ee6c81e403cb17fca3b2be749352a6\": rpc error: code = NotFound desc = could not find container \"2f56b2b956829321f0db5bdfc92578e388ee6c81e403cb17fca3b2be749352a6\": container with ID starting with 2f56b2b956829321f0db5bdfc92578e388ee6c81e403cb17fca3b2be749352a6 not found: ID does not exist" Jul 29 01:26:08.660116 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.660078 2543 scope.go:117] "RemoveContainer" containerID="54feb74a79c4d199ff6b822eb863f2986d8e1789287c2e38fd7f1b953dbbc6e0" Jul 29 01:26:08.660341 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:26:08.660323 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54feb74a79c4d199ff6b822eb863f2986d8e1789287c2e38fd7f1b953dbbc6e0\": container with ID starting with 54feb74a79c4d199ff6b822eb863f2986d8e1789287c2e38fd7f1b953dbbc6e0 not found: ID does not exist" containerID="54feb74a79c4d199ff6b822eb863f2986d8e1789287c2e38fd7f1b953dbbc6e0" Jul 29 01:26:08.660379 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.660349 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54feb74a79c4d199ff6b822eb863f2986d8e1789287c2e38fd7f1b953dbbc6e0"} err="failed to get container status \"54feb74a79c4d199ff6b822eb863f2986d8e1789287c2e38fd7f1b953dbbc6e0\": rpc error: code = NotFound desc = could not find container \"54feb74a79c4d199ff6b822eb863f2986d8e1789287c2e38fd7f1b953dbbc6e0\": container with ID starting with 54feb74a79c4d199ff6b822eb863f2986d8e1789287c2e38fd7f1b953dbbc6e0 not found: ID does not exist" Jul 29 01:26:08.663522 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.663500 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq"] Jul 29 01:26:08.667325 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:08.667294 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-bb686069-openshift-default-6649bf4cf4-967nq"] Jul 29 01:26:10.343388 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:10.343355 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" path="/var/lib/kubelet/pods/9c0095ba-60b3-4ee1-8fdb-a756bc9edc65/volumes" Jul 29 01:26:21.281914 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:21.281062 2543 patch_prober.go:28] interesting pod/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": read tcp 10.132.0.2:43796->10.132.0.36:15021: read: connection reset by peer" start-of-body= Jul 29 01:26:21.281914 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:21.281127 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" podUID="0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": read tcp 10.132.0.2:43796->10.132.0.36:15021: read: connection reset by peer" Jul 29 01:26:21.713621 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:21.713590 2543 generic.go:356] "Generic (PLEG): container finished" podID="0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" containerID="758ec61d92183c7bebcc9c9e0bb1d0a41684512c3d095a3bd33838b8180b10f9" exitCode=0 Jul 29 01:26:21.713799 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:21.713638 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" event={"ID":"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8","Type":"ContainerDied","Data":"758ec61d92183c7bebcc9c9e0bb1d0a41684512c3d095a3bd33838b8180b10f9"} Jul 29 01:26:21.714013 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:21.714001 2543 scope.go:117] "RemoveContainer" containerID="758ec61d92183c7bebcc9c9e0bb1d0a41684512c3d095a3bd33838b8180b10f9" Jul 29 01:26:21.714988 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:21.714960 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:26:21.715094 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:21.715042 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:26:21.715094 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:21.715089 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:26:22.720608 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:22.720574 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" event={"ID":"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8","Type":"ContainerStarted","Data":"c6cdd4e92ae05f298526e891afcccc13aeb4d3eaecb814e46f92e42cee7944f9"} Jul 29 01:26:23.365296 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:23.365260 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:23.365478 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:23.365456 2543 patch_prober.go:28] interesting pod/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 29 01:26:23.365527 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:23.365503 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" podUID="0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 29 01:26:24.364928 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:24.364890 2543 patch_prober.go:28] interesting pod/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 29 01:26:24.365307 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:24.364947 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" podUID="0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 29 01:26:25.365720 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:25.365681 2543 patch_prober.go:28] interesting pod/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 29 01:26:25.366148 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:25.365735 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" podUID="0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 29 01:26:26.365243 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:26.365208 2543 patch_prober.go:28] interesting pod/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 29 01:26:26.365426 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:26.365257 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" podUID="0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 29 01:26:27.368936 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:27.368903 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:27.369424 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:27.369280 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:27.369988 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:27.369967 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:31.085844 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:31.085796 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aaa0c308-x5n92"] Jul 29 01:26:31.086221 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:31.086146 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" containerName="istio-proxy" Jul 29 01:26:31.086221 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:31.086159 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" containerName="istio-proxy" Jul 29 01:26:31.086221 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:31.086175 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" containerName="istio-proxy" Jul 29 01:26:31.086221 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:31.086180 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" containerName="istio-proxy" Jul 29 01:26:31.086410 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:31.086231 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" containerName="istio-proxy" Jul 29 01:26:31.086410 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:31.086240 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c0095ba-60b3-4ee1-8fdb-a756bc9edc65" containerName="istio-proxy" Jul 29 01:26:31.093278 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:31.093257 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aaa0c308-x5n92" Jul 29 01:26:31.095690 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:31.095658 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 01:26:31.096024 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:31.095919 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjvns\"" Jul 29 01:26:31.096896 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:31.096876 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aaa0c308-x5n92"] Jul 29 01:26:31.154314 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:31.154282 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dedee25-30e9-4368-9b35-c661bbfc0880-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-aaa0c308-x5n92\" (UID: \"2dedee25-30e9-4368-9b35-c661bbfc0880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aaa0c308-x5n92" Jul 29 01:26:31.255761 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:31.255724 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dedee25-30e9-4368-9b35-c661bbfc0880-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-aaa0c308-x5n92\" (UID: \"2dedee25-30e9-4368-9b35-c661bbfc0880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aaa0c308-x5n92" Jul 29 01:26:31.256423 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:31.256397 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dedee25-30e9-4368-9b35-c661bbfc0880-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-aaa0c308-x5n92\" (UID: \"2dedee25-30e9-4368-9b35-c661bbfc0880\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aaa0c308-x5n92" Jul 29 01:26:31.408967 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:31.408876 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aaa0c308-x5n92" Jul 29 01:26:31.757991 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:31.757914 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aaa0c308-x5n92"] Jul 29 01:26:31.765011 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:26:31.764985 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dedee25_30e9_4368_9b35_c661bbfc0880.slice/crio-4ff67050337b644d48eed4c05fdea8ac9610551d7de43eaa9151d225f394c787 WatchSource:0}: Error finding container 4ff67050337b644d48eed4c05fdea8ac9610551d7de43eaa9151d225f394c787: Status 404 returned error can't find the container with id 4ff67050337b644d48eed4c05fdea8ac9610551d7de43eaa9151d225f394c787 Jul 29 01:26:32.767952 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:32.767857 2543 generic.go:356] "Generic (PLEG): container finished" podID="2dedee25-30e9-4368-9b35-c661bbfc0880" containerID="7b2e12f2a644b574699b9eb5f9081f25bff60515ca743f546022e6b29d4474ac" exitCode=0 Jul 29 01:26:32.768305 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:32.767993 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aaa0c308-x5n92" event={"ID":"2dedee25-30e9-4368-9b35-c661bbfc0880","Type":"ContainerDied","Data":"7b2e12f2a644b574699b9eb5f9081f25bff60515ca743f546022e6b29d4474ac"} Jul 29 01:26:32.768305 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:32.768053 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aaa0c308-x5n92" event={"ID":"2dedee25-30e9-4368-9b35-c661bbfc0880","Type":"ContainerStarted","Data":"4ff67050337b644d48eed4c05fdea8ac9610551d7de43eaa9151d225f394c787"} Jul 29 01:26:33.911961 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:33.911935 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aaa0c308-x5n92" Jul 29 01:26:33.982708 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:33.982668 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dedee25-30e9-4368-9b35-c661bbfc0880-maas-api-service-ca\") pod \"2dedee25-30e9-4368-9b35-c661bbfc0880\" (UID: \"2dedee25-30e9-4368-9b35-c661bbfc0880\") " Jul 29 01:26:33.983058 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:33.983033 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dedee25-30e9-4368-9b35-c661bbfc0880-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2dedee25-30e9-4368-9b35-c661bbfc0880" (UID: "2dedee25-30e9-4368-9b35-c661bbfc0880"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:26:34.084253 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:34.084216 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dedee25-30e9-4368-9b35-c661bbfc0880-maas-api-service-ca\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:26:34.777777 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:34.777692 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aaa0c308-x5n92" event={"ID":"2dedee25-30e9-4368-9b35-c661bbfc0880","Type":"ContainerDied","Data":"4ff67050337b644d48eed4c05fdea8ac9610551d7de43eaa9151d225f394c787"} Jul 29 01:26:34.777777 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:34.777747 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ff67050337b644d48eed4c05fdea8ac9610551d7de43eaa9151d225f394c787" Jul 29 01:26:34.777777 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:34.777706 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aaa0c308-x5n92" Jul 29 01:26:37.304572 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:37.304530 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aaa0c308-x5n92"] Jul 29 01:26:37.306850 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:37.306812 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-aaa0c308-x5n92"] Jul 29 01:26:37.434429 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:37.434390 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm"] Jul 29 01:26:37.434655 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:37.434626 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" podUID="0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" containerName="istio-proxy" containerID="cri-o://c6cdd4e92ae05f298526e891afcccc13aeb4d3eaecb814e46f92e42cee7944f9" gracePeriod=30 Jul 29 01:26:38.194866 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.194834 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q"] Jul 29 01:26:38.195229 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.195209 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dedee25-30e9-4368-9b35-c661bbfc0880" containerName="revoke-keys" Jul 29 01:26:38.195274 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.195231 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dedee25-30e9-4368-9b35-c661bbfc0880" containerName="revoke-keys" Jul 29 01:26:38.195312 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.195295 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dedee25-30e9-4368-9b35-c661bbfc0880" containerName="revoke-keys" Jul 29 01:26:38.198472 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.198453 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.200795 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.200774 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8c8da55e-openshift-default-dockercfg-ns9f6\"" Jul 29 01:26:38.210095 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.210073 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q"] Jul 29 01:26:38.322869 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.322809 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6c2964-4fa6-4944-b083-b48e72f3c110-istiod-ca-cert\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.322869 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.322873 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-credential-socket\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.323362 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.322909 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-envoy\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.323362 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.322929 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-token\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.323362 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.323041 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-workload-socket\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.323362 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.323075 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-workload-certs\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.323362 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.323120 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpmb7\" (UniqueName: \"kubernetes.io/projected/9c6c2964-4fa6-4944-b083-b48e72f3c110-kube-api-access-kpmb7\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.323362 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.323185 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-podinfo\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.323362 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.323232 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-data\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.342755 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.342717 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dedee25-30e9-4368-9b35-c661bbfc0880" path="/var/lib/kubelet/pods/2dedee25-30e9-4368-9b35-c661bbfc0880/volumes" Jul 29 01:26:38.424438 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.424396 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpmb7\" (UniqueName: \"kubernetes.io/projected/9c6c2964-4fa6-4944-b083-b48e72f3c110-kube-api-access-kpmb7\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.424617 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.424480 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-podinfo\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.424617 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.424517 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-data\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.424617 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.424548 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6c2964-4fa6-4944-b083-b48e72f3c110-istiod-ca-cert\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.424617 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.424575 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-credential-socket\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.424890 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.424623 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-envoy\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.424890 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.424648 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-token\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.424890 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.424733 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-workload-socket\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.424890 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.424760 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-workload-certs\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.425105 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.425063 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-credential-socket\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.425191 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.425169 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-data\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.425284 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.425252 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-workload-socket\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.425427 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.425317 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-workload-certs\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.426035 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.426007 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6c2964-4fa6-4944-b083-b48e72f3c110-istiod-ca-cert\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.427075 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.427056 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-envoy\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.427300 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.427284 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-podinfo\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.432533 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.432507 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpmb7\" (UniqueName: \"kubernetes.io/projected/9c6c2964-4fa6-4944-b083-b48e72f3c110-kube-api-access-kpmb7\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.432867 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.432842 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-token\") pod \"e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.509399 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.509286 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:38.644727 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.644703 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q"] Jul 29 01:26:38.646449 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:26:38.646424 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6c2964_4fa6_4944_b083_b48e72f3c110.slice/crio-820c1ec8642867ca80e26c42f6cbb4d0ad35b292b7b5ee9893c35178144a37af WatchSource:0}: Error finding container 820c1ec8642867ca80e26c42f6cbb4d0ad35b292b7b5ee9893c35178144a37af: Status 404 returned error can't find the container with id 820c1ec8642867ca80e26c42f6cbb4d0ad35b292b7b5ee9893c35178144a37af Jul 29 01:26:38.648541 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.648512 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:26:38.648599 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.648577 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:26:38.648640 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.648602 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:26:38.798303 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.798216 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" event={"ID":"9c6c2964-4fa6-4944-b083-b48e72f3c110","Type":"ContainerStarted","Data":"af5b80cc7b3d77e23e8b11c90e3725618fdf69d76ec9d0abc150ca5c9bd8469f"} Jul 29 01:26:38.798303 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.798272 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" event={"ID":"9c6c2964-4fa6-4944-b083-b48e72f3c110","Type":"ContainerStarted","Data":"820c1ec8642867ca80e26c42f6cbb4d0ad35b292b7b5ee9893c35178144a37af"} Jul 29 01:26:38.821761 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:38.821696 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" podStartSLOduration=0.821673188 podStartE2EDuration="821.673188ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:26:38.818285986 +0000 UTC m=+1949.088930310" watchObservedRunningTime="2026-07-29 01:26:38.821673188 +0000 UTC m=+1949.092317495" Jul 29 01:26:39.510196 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:39.510156 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:39.515282 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:39.515261 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:39.803353 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:39.803274 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:39.804123 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:39.804103 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:42.371706 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.371674 2543 patch_prober.go:28] interesting pod/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 01:26:42.372087 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.371727 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" podUID="0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 01:26:42.688708 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.688681 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:42.762052 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.762016 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-workload-socket\") pod \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " Jul 29 01:26:42.762052 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.762057 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-token\") pod \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " Jul 29 01:26:42.762256 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.762074 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istiod-ca-cert\") pod \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " Jul 29 01:26:42.762256 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.762097 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztj7w\" (UniqueName: \"kubernetes.io/projected/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-kube-api-access-ztj7w\") pod \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " Jul 29 01:26:42.762331 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.762265 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-envoy\") pod \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " Jul 29 01:26:42.762331 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.762325 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-podinfo\") pod \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " Jul 29 01:26:42.762432 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.762332 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" (UID: "0a5bd9b4-3986-416d-bdef-0b3b3143bfe8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:26:42.762432 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.762388 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-workload-certs\") pod \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " Jul 29 01:26:42.762432 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.762417 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-data\") pod \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " Jul 29 01:26:42.762587 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.762463 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-credential-socket\") pod \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\" (UID: \"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8\") " Jul 29 01:26:42.762587 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.762487 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" (UID: "0a5bd9b4-3986-416d-bdef-0b3b3143bfe8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:26:42.762703 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.762628 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" (UID: "0a5bd9b4-3986-416d-bdef-0b3b3143bfe8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:26:42.762863 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.762840 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-workload-socket\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:26:42.762972 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.762869 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istiod-ca-cert\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:26:42.762972 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.762883 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-workload-certs\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:26:42.762972 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.762896 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" (UID: "0a5bd9b4-3986-416d-bdef-0b3b3143bfe8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:26:42.763436 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.763411 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-data" (OuterVolumeSpecName: "istio-data") pod "0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" (UID: "0a5bd9b4-3986-416d-bdef-0b3b3143bfe8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:26:42.764473 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.764446 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-kube-api-access-ztj7w" (OuterVolumeSpecName: "kube-api-access-ztj7w") pod "0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" (UID: "0a5bd9b4-3986-416d-bdef-0b3b3143bfe8"). InnerVolumeSpecName "kube-api-access-ztj7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:26:42.764568 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.764491 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-token" (OuterVolumeSpecName: "istio-token") pod "0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" (UID: "0a5bd9b4-3986-416d-bdef-0b3b3143bfe8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:26:42.764568 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.764523 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" (UID: "0a5bd9b4-3986-416d-bdef-0b3b3143bfe8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 01:26:42.764642 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.764563 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" (UID: "0a5bd9b4-3986-416d-bdef-0b3b3143bfe8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:26:42.821340 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.821301 2543 generic.go:356] "Generic (PLEG): container finished" podID="0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" containerID="c6cdd4e92ae05f298526e891afcccc13aeb4d3eaecb814e46f92e42cee7944f9" exitCode=0 Jul 29 01:26:42.821532 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.821356 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" Jul 29 01:26:42.821532 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.821388 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" event={"ID":"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8","Type":"ContainerDied","Data":"c6cdd4e92ae05f298526e891afcccc13aeb4d3eaecb814e46f92e42cee7944f9"} Jul 29 01:26:42.821532 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.821446 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm" event={"ID":"0a5bd9b4-3986-416d-bdef-0b3b3143bfe8","Type":"ContainerDied","Data":"82d27beb8dfa4769a8809f3570e3180368456b4f4b2cd1bcd78280d327f0a283"} Jul 29 01:26:42.821532 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.821467 2543 scope.go:117] "RemoveContainer" containerID="c6cdd4e92ae05f298526e891afcccc13aeb4d3eaecb814e46f92e42cee7944f9" Jul 29 01:26:42.830262 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.830243 2543 scope.go:117] "RemoveContainer" containerID="758ec61d92183c7bebcc9c9e0bb1d0a41684512c3d095a3bd33838b8180b10f9" Jul 29 01:26:42.839300 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.839277 2543 scope.go:117] "RemoveContainer" containerID="c6cdd4e92ae05f298526e891afcccc13aeb4d3eaecb814e46f92e42cee7944f9" Jul 29 01:26:42.839564 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:26:42.839543 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6cdd4e92ae05f298526e891afcccc13aeb4d3eaecb814e46f92e42cee7944f9\": container with ID starting with c6cdd4e92ae05f298526e891afcccc13aeb4d3eaecb814e46f92e42cee7944f9 not found: ID does not exist" containerID="c6cdd4e92ae05f298526e891afcccc13aeb4d3eaecb814e46f92e42cee7944f9" Jul 29 01:26:42.839637 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.839572 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6cdd4e92ae05f298526e891afcccc13aeb4d3eaecb814e46f92e42cee7944f9"} err="failed to get container status \"c6cdd4e92ae05f298526e891afcccc13aeb4d3eaecb814e46f92e42cee7944f9\": rpc error: code = NotFound desc = could not find container \"c6cdd4e92ae05f298526e891afcccc13aeb4d3eaecb814e46f92e42cee7944f9\": container with ID starting with c6cdd4e92ae05f298526e891afcccc13aeb4d3eaecb814e46f92e42cee7944f9 not found: ID does not exist" Jul 29 01:26:42.839637 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.839589 2543 scope.go:117] "RemoveContainer" containerID="758ec61d92183c7bebcc9c9e0bb1d0a41684512c3d095a3bd33838b8180b10f9" Jul 29 01:26:42.839883 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:26:42.839853 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"758ec61d92183c7bebcc9c9e0bb1d0a41684512c3d095a3bd33838b8180b10f9\": container with ID starting with 758ec61d92183c7bebcc9c9e0bb1d0a41684512c3d095a3bd33838b8180b10f9 not found: ID does not exist" containerID="758ec61d92183c7bebcc9c9e0bb1d0a41684512c3d095a3bd33838b8180b10f9" Jul 29 01:26:42.840016 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.839885 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"758ec61d92183c7bebcc9c9e0bb1d0a41684512c3d095a3bd33838b8180b10f9"} err="failed to get container status \"758ec61d92183c7bebcc9c9e0bb1d0a41684512c3d095a3bd33838b8180b10f9\": rpc error: code = NotFound desc = could not find container \"758ec61d92183c7bebcc9c9e0bb1d0a41684512c3d095a3bd33838b8180b10f9\": container with ID starting with 758ec61d92183c7bebcc9c9e0bb1d0a41684512c3d095a3bd33838b8180b10f9 not found: ID does not exist" Jul 29 01:26:42.842443 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.842425 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm"] Jul 29 01:26:42.845937 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.845918 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-aaa0c308-gw-openshift-default-7848fc58d4-25jgm"] Jul 29 01:26:42.863315 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.863288 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-credential-socket\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:26:42.863315 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.863313 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-token\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:26:42.863453 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.863324 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztj7w\" (UniqueName: \"kubernetes.io/projected/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-kube-api-access-ztj7w\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:26:42.863453 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.863334 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-envoy\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:26:42.863453 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.863343 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-podinfo\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:26:42.863453 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:42.863351 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8-istio-data\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:26:44.343775 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:44.343743 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" path="/var/lib/kubelet/pods/0a5bd9b4-3986-416d-bdef-0b3b3143bfe8/volumes" Jul 29 01:26:55.345233 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:55.345199 2543 patch_prober.go:28] interesting pod/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": read tcp 10.132.0.2:52736->10.132.0.38:15021: read: connection reset by peer" start-of-body= Jul 29 01:26:55.345630 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:55.345317 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" podUID="9c6c2964-4fa6-4944-b083-b48e72f3c110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": read tcp 10.132.0.2:52736->10.132.0.38:15021: read: connection reset by peer" Jul 29 01:26:55.883736 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:55.883704 2543 generic.go:356] "Generic (PLEG): container finished" podID="9c6c2964-4fa6-4944-b083-b48e72f3c110" containerID="af5b80cc7b3d77e23e8b11c90e3725618fdf69d76ec9d0abc150ca5c9bd8469f" exitCode=0 Jul 29 01:26:55.883906 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:55.883773 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" event={"ID":"9c6c2964-4fa6-4944-b083-b48e72f3c110","Type":"ContainerDied","Data":"af5b80cc7b3d77e23e8b11c90e3725618fdf69d76ec9d0abc150ca5c9bd8469f"} Jul 29 01:26:55.884181 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:55.884168 2543 scope.go:117] "RemoveContainer" containerID="af5b80cc7b3d77e23e8b11c90e3725618fdf69d76ec9d0abc150ca5c9bd8469f" Jul 29 01:26:55.885253 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:55.885227 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:26:55.885324 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:55.885306 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:26:55.885370 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:55.885355 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:26:56.891165 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:56.891116 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" event={"ID":"9c6c2964-4fa6-4944-b083-b48e72f3c110","Type":"ContainerStarted","Data":"31c7b4a0ce75d2011e6d8bac48a41325b88cdbf3617c5c935c94eb2392a6745d"} Jul 29 01:26:57.510251 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:57.510216 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:26:57.510612 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:57.510589 2543 patch_prober.go:28] interesting pod/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 29 01:26:57.510774 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:57.510752 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" podUID="9c6c2964-4fa6-4944-b083-b48e72f3c110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 29 01:26:58.511386 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:58.511343 2543 patch_prober.go:28] interesting pod/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 29 01:26:58.511926 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:58.511406 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" podUID="9c6c2964-4fa6-4944-b083-b48e72f3c110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 29 01:26:59.510011 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:59.509971 2543 patch_prober.go:28] interesting pod/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 29 01:26:59.510299 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:26:59.510269 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" podUID="9c6c2964-4fa6-4944-b083-b48e72f3c110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 29 01:27:00.510526 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:00.510479 2543 patch_prober.go:28] interesting pod/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Jul 29 01:27:00.511054 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:00.510559 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" podUID="9c6c2964-4fa6-4944-b083-b48e72f3c110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Jul 29 01:27:01.513553 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:01.513521 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:27:01.514046 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:01.513920 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:27:01.514558 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:01.514538 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:27:05.880620 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:05.880543 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c8da55e-sk9kt"] Jul 29 01:27:05.880994 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:05.880933 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" containerName="istio-proxy" Jul 29 01:27:05.880994 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:05.880945 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" containerName="istio-proxy" Jul 29 01:27:05.880994 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:05.880954 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" containerName="istio-proxy" Jul 29 01:27:05.880994 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:05.880960 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" containerName="istio-proxy" Jul 29 01:27:05.881130 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:05.881011 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" containerName="istio-proxy" Jul 29 01:27:05.883143 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:05.883128 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c8da55e-sk9kt" Jul 29 01:27:05.885409 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:05.885391 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 01:27:05.885555 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:05.885532 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjvns\"" Jul 29 01:27:05.890653 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:05.890631 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c8da55e-sk9kt"] Jul 29 01:27:05.973355 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:05.973305 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acc01f10-4e5b-40b8-b2ae-55fa42230a2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c8da55e-sk9kt\" (UID: \"acc01f10-4e5b-40b8-b2ae-55fa42230a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c8da55e-sk9kt" Jul 29 01:27:06.073750 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:06.073707 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acc01f10-4e5b-40b8-b2ae-55fa42230a2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c8da55e-sk9kt\" (UID: \"acc01f10-4e5b-40b8-b2ae-55fa42230a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c8da55e-sk9kt" Jul 29 01:27:06.074429 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:06.074405 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acc01f10-4e5b-40b8-b2ae-55fa42230a2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c8da55e-sk9kt\" (UID: \"acc01f10-4e5b-40b8-b2ae-55fa42230a2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c8da55e-sk9kt" Jul 29 01:27:06.194496 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:06.194409 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c8da55e-sk9kt" Jul 29 01:27:06.323905 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:06.323878 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c8da55e-sk9kt"] Jul 29 01:27:06.325916 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:27:06.325876 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc01f10_4e5b_40b8_b2ae_55fa42230a2c.slice/crio-3759f303345d00094773a201d84f74e8a905efed24afa5ee9f5a73ae9bb4d470 WatchSource:0}: Error finding container 3759f303345d00094773a201d84f74e8a905efed24afa5ee9f5a73ae9bb4d470: Status 404 returned error can't find the container with id 3759f303345d00094773a201d84f74e8a905efed24afa5ee9f5a73ae9bb4d470 Jul 29 01:27:06.950769 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:06.950732 2543 generic.go:356] "Generic (PLEG): container finished" podID="acc01f10-4e5b-40b8-b2ae-55fa42230a2c" containerID="01e6ae11b3143293734693f1c6ce2aab073d74d8a49045bec9b8c755d483a00e" exitCode=0 Jul 29 01:27:06.951131 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:06.950835 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c8da55e-sk9kt" event={"ID":"acc01f10-4e5b-40b8-b2ae-55fa42230a2c","Type":"ContainerDied","Data":"01e6ae11b3143293734693f1c6ce2aab073d74d8a49045bec9b8c755d483a00e"} Jul 29 01:27:06.951131 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:06.950893 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c8da55e-sk9kt" event={"ID":"acc01f10-4e5b-40b8-b2ae-55fa42230a2c","Type":"ContainerStarted","Data":"3759f303345d00094773a201d84f74e8a905efed24afa5ee9f5a73ae9bb4d470"} Jul 29 01:27:08.083651 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:08.083625 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c8da55e-sk9kt" Jul 29 01:27:08.189837 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:08.189773 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acc01f10-4e5b-40b8-b2ae-55fa42230a2c-maas-api-service-ca\") pod \"acc01f10-4e5b-40b8-b2ae-55fa42230a2c\" (UID: \"acc01f10-4e5b-40b8-b2ae-55fa42230a2c\") " Jul 29 01:27:08.190134 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:08.190110 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc01f10-4e5b-40b8-b2ae-55fa42230a2c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "acc01f10-4e5b-40b8-b2ae-55fa42230a2c" (UID: "acc01f10-4e5b-40b8-b2ae-55fa42230a2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:27:08.290851 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:08.290752 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acc01f10-4e5b-40b8-b2ae-55fa42230a2c-maas-api-service-ca\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:27:08.961591 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:08.961560 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c8da55e-sk9kt" Jul 29 01:27:08.961591 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:08.961586 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c8da55e-sk9kt" event={"ID":"acc01f10-4e5b-40b8-b2ae-55fa42230a2c","Type":"ContainerDied","Data":"3759f303345d00094773a201d84f74e8a905efed24afa5ee9f5a73ae9bb4d470"} Jul 29 01:27:08.961799 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:08.961632 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3759f303345d00094773a201d84f74e8a905efed24afa5ee9f5a73ae9bb4d470" Jul 29 01:27:11.137791 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:11.137758 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c8da55e-sk9kt"] Jul 29 01:27:11.140157 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:11.140136 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c8da55e-sk9kt"] Jul 29 01:27:12.244721 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:12.244685 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q"] Jul 29 01:27:12.245200 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:12.245019 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" podUID="9c6c2964-4fa6-4944-b083-b48e72f3c110" containerName="istio-proxy" containerID="cri-o://31c7b4a0ce75d2011e6d8bac48a41325b88cdbf3617c5c935c94eb2392a6745d" gracePeriod=30 Jul 29 01:27:12.342921 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:12.342879 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acc01f10-4e5b-40b8-b2ae-55fa42230a2c" path="/var/lib/kubelet/pods/acc01f10-4e5b-40b8-b2ae-55fa42230a2c/volumes" Jul 29 01:27:12.965408 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:12.965378 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb"] Jul 29 01:27:12.965740 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:12.965728 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acc01f10-4e5b-40b8-b2ae-55fa42230a2c" containerName="revoke-keys" Jul 29 01:27:12.965784 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:12.965741 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="acc01f10-4e5b-40b8-b2ae-55fa42230a2c" containerName="revoke-keys" Jul 29 01:27:12.965845 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:12.965835 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a5bd9b4-3986-416d-bdef-0b3b3143bfe8" containerName="istio-proxy" Jul 29 01:27:12.965887 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:12.965845 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="acc01f10-4e5b-40b8-b2ae-55fa42230a2c" containerName="revoke-keys" Jul 29 01:27:12.968394 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:12.968374 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:12.970572 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:12.970555 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c28408bd-openshift-default-dockercfg-hcf2w\"" Jul 29 01:27:12.982023 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:12.981998 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb"] Jul 29 01:27:13.030611 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.030570 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-workload-certs\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.030611 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.030604 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-envoy\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.030863 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.030623 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istiod-ca-cert\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.030863 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.030709 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-workload-socket\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.030863 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.030751 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-token\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.030863 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.030775 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-data\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.030863 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.030854 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-podinfo\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.031141 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.030888 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-credential-socket\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.031141 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.030913 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n548s\" (UniqueName: \"kubernetes.io/projected/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-kube-api-access-n548s\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.131926 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.131886 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-podinfo\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.131926 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.131929 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-credential-socket\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.132176 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.131956 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n548s\" (UniqueName: \"kubernetes.io/projected/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-kube-api-access-n548s\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.132176 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.131996 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-workload-certs\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.132176 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.132023 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-envoy\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.132176 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.132044 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istiod-ca-cert\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.132176 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.132112 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-workload-socket\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.132176 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.132162 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-token\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.132679 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.132198 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-data\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.132679 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.132373 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-workload-certs\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.132679 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.132404 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-credential-socket\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.132867 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.132690 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-workload-socket\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.132867 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.132780 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-data\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.132867 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.132789 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istiod-ca-cert\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.134809 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.134787 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-envoy\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.134979 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.134963 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-podinfo\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.139429 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.139407 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n548s\" (UniqueName: \"kubernetes.io/projected/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-kube-api-access-n548s\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.140014 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.139994 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-token\") pod \"e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.278397 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.278305 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:13.407233 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.407209 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb"] Jul 29 01:27:13.409124 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:27:13.409100 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd779a0e7_7e7a_4219_ad72_3c48f1480cc6.slice/crio-beea81e567e636dfb78c2d0108556c322d043d6a3748e07f185a5b210bf87493 WatchSource:0}: Error finding container beea81e567e636dfb78c2d0108556c322d043d6a3748e07f185a5b210bf87493: Status 404 returned error can't find the container with id beea81e567e636dfb78c2d0108556c322d043d6a3748e07f185a5b210bf87493 Jul 29 01:27:13.411207 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.411179 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:27:13.411285 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.411239 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:27:13.411285 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.411277 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:27:13.988480 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.988442 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" event={"ID":"d779a0e7-7e7a-4219-ad72-3c48f1480cc6","Type":"ContainerStarted","Data":"8534f0095d89b01b8c20561f210d2a9bde433940bcf4b289ce9beab37591dd3b"} Jul 29 01:27:13.988679 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:13.988493 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" event={"ID":"d779a0e7-7e7a-4219-ad72-3c48f1480cc6","Type":"ContainerStarted","Data":"beea81e567e636dfb78c2d0108556c322d043d6a3748e07f185a5b210bf87493"} Jul 29 01:27:14.030414 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:14.030358 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" podStartSLOduration=2.030343431 podStartE2EDuration="2.030343431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:27:14.02711176 +0000 UTC m=+1984.297756066" watchObservedRunningTime="2026-07-29 01:27:14.030343431 +0000 UTC m=+1984.300987738" Jul 29 01:27:14.279119 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:14.279033 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:14.284184 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:14.284162 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:14.992914 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:14.992880 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:14.993718 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:14.993700 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:16.516279 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:16.516244 2543 patch_prober.go:28] interesting pod/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 01:27:16.516651 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:16.516299 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" podUID="9c6c2964-4fa6-4944-b083-b48e72f3c110" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 01:27:17.493524 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.493503 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:27:17.577000 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.576968 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-data\") pod \"9c6c2964-4fa6-4944-b083-b48e72f3c110\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " Jul 29 01:27:17.577464 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.577012 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-workload-socket\") pod \"9c6c2964-4fa6-4944-b083-b48e72f3c110\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " Jul 29 01:27:17.577464 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.577044 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-token\") pod \"9c6c2964-4fa6-4944-b083-b48e72f3c110\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " Jul 29 01:27:17.577464 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.577069 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-envoy\") pod \"9c6c2964-4fa6-4944-b083-b48e72f3c110\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " Jul 29 01:27:17.577464 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.577108 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpmb7\" (UniqueName: \"kubernetes.io/projected/9c6c2964-4fa6-4944-b083-b48e72f3c110-kube-api-access-kpmb7\") pod \"9c6c2964-4fa6-4944-b083-b48e72f3c110\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " Jul 29 01:27:17.577464 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.577138 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-workload-certs\") pod \"9c6c2964-4fa6-4944-b083-b48e72f3c110\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " Jul 29 01:27:17.577464 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.577210 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6c2964-4fa6-4944-b083-b48e72f3c110-istiod-ca-cert\") pod \"9c6c2964-4fa6-4944-b083-b48e72f3c110\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " Jul 29 01:27:17.577464 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.577234 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-credential-socket\") pod \"9c6c2964-4fa6-4944-b083-b48e72f3c110\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " Jul 29 01:27:17.577464 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.577263 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-podinfo\") pod \"9c6c2964-4fa6-4944-b083-b48e72f3c110\" (UID: \"9c6c2964-4fa6-4944-b083-b48e72f3c110\") " Jul 29 01:27:17.577464 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.577279 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9c6c2964-4fa6-4944-b083-b48e72f3c110" (UID: "9c6c2964-4fa6-4944-b083-b48e72f3c110"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:27:17.577977 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.577493 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9c6c2964-4fa6-4944-b083-b48e72f3c110" (UID: "9c6c2964-4fa6-4944-b083-b48e72f3c110"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:27:17.577977 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.577581 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c6c2964-4fa6-4944-b083-b48e72f3c110-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9c6c2964-4fa6-4944-b083-b48e72f3c110" (UID: "9c6c2964-4fa6-4944-b083-b48e72f3c110"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:27:17.577977 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.577695 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6c2964-4fa6-4944-b083-b48e72f3c110-istiod-ca-cert\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:27:17.577977 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.577712 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-credential-socket\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:27:17.577977 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.577727 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-workload-socket\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:27:17.577977 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.577953 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-data" (OuterVolumeSpecName: "istio-data") pod "9c6c2964-4fa6-4944-b083-b48e72f3c110" (UID: "9c6c2964-4fa6-4944-b083-b48e72f3c110"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:27:17.578178 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.578015 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9c6c2964-4fa6-4944-b083-b48e72f3c110" (UID: "9c6c2964-4fa6-4944-b083-b48e72f3c110"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:27:17.579782 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.579760 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-token" (OuterVolumeSpecName: "istio-token") pod "9c6c2964-4fa6-4944-b083-b48e72f3c110" (UID: "9c6c2964-4fa6-4944-b083-b48e72f3c110"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:27:17.579885 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.579796 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9c6c2964-4fa6-4944-b083-b48e72f3c110" (UID: "9c6c2964-4fa6-4944-b083-b48e72f3c110"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:27:17.579885 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.579803 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9c6c2964-4fa6-4944-b083-b48e72f3c110" (UID: "9c6c2964-4fa6-4944-b083-b48e72f3c110"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 01:27:17.579885 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.579851 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6c2964-4fa6-4944-b083-b48e72f3c110-kube-api-access-kpmb7" (OuterVolumeSpecName: "kube-api-access-kpmb7") pod "9c6c2964-4fa6-4944-b083-b48e72f3c110" (UID: "9c6c2964-4fa6-4944-b083-b48e72f3c110"). InnerVolumeSpecName "kube-api-access-kpmb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:27:17.678622 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.678591 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-envoy\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:27:17.678622 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.678617 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpmb7\" (UniqueName: \"kubernetes.io/projected/9c6c2964-4fa6-4944-b083-b48e72f3c110-kube-api-access-kpmb7\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:27:17.678622 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.678626 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-workload-certs\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:27:17.678882 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.678638 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-podinfo\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:27:17.678882 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.678645 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-data\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:27:17.678882 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:17.678654 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6c2964-4fa6-4944-b083-b48e72f3c110-istio-token\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:27:18.010399 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:18.010305 2543 generic.go:356] "Generic (PLEG): container finished" podID="9c6c2964-4fa6-4944-b083-b48e72f3c110" containerID="31c7b4a0ce75d2011e6d8bac48a41325b88cdbf3617c5c935c94eb2392a6745d" exitCode=0 Jul 29 01:27:18.010399 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:18.010350 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" Jul 29 01:27:18.010578 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:18.010397 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" event={"ID":"9c6c2964-4fa6-4944-b083-b48e72f3c110","Type":"ContainerDied","Data":"31c7b4a0ce75d2011e6d8bac48a41325b88cdbf3617c5c935c94eb2392a6745d"} Jul 29 01:27:18.010578 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:18.010457 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q" event={"ID":"9c6c2964-4fa6-4944-b083-b48e72f3c110","Type":"ContainerDied","Data":"820c1ec8642867ca80e26c42f6cbb4d0ad35b292b7b5ee9893c35178144a37af"} Jul 29 01:27:18.010578 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:18.010487 2543 scope.go:117] "RemoveContainer" containerID="31c7b4a0ce75d2011e6d8bac48a41325b88cdbf3617c5c935c94eb2392a6745d" Jul 29 01:27:18.019769 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:18.019752 2543 scope.go:117] "RemoveContainer" containerID="af5b80cc7b3d77e23e8b11c90e3725618fdf69d76ec9d0abc150ca5c9bd8469f" Jul 29 01:27:18.030302 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:18.030283 2543 scope.go:117] "RemoveContainer" containerID="31c7b4a0ce75d2011e6d8bac48a41325b88cdbf3617c5c935c94eb2392a6745d" Jul 29 01:27:18.030647 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:27:18.030601 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31c7b4a0ce75d2011e6d8bac48a41325b88cdbf3617c5c935c94eb2392a6745d\": container with ID starting with 31c7b4a0ce75d2011e6d8bac48a41325b88cdbf3617c5c935c94eb2392a6745d not found: ID does not exist" containerID="31c7b4a0ce75d2011e6d8bac48a41325b88cdbf3617c5c935c94eb2392a6745d" Jul 29 01:27:18.030739 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:18.030636 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31c7b4a0ce75d2011e6d8bac48a41325b88cdbf3617c5c935c94eb2392a6745d"} err="failed to get container status \"31c7b4a0ce75d2011e6d8bac48a41325b88cdbf3617c5c935c94eb2392a6745d\": rpc error: code = NotFound desc = could not find container \"31c7b4a0ce75d2011e6d8bac48a41325b88cdbf3617c5c935c94eb2392a6745d\": container with ID starting with 31c7b4a0ce75d2011e6d8bac48a41325b88cdbf3617c5c935c94eb2392a6745d not found: ID does not exist" Jul 29 01:27:18.030739 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:18.030661 2543 scope.go:117] "RemoveContainer" containerID="af5b80cc7b3d77e23e8b11c90e3725618fdf69d76ec9d0abc150ca5c9bd8469f" Jul 29 01:27:18.030960 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:27:18.030943 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af5b80cc7b3d77e23e8b11c90e3725618fdf69d76ec9d0abc150ca5c9bd8469f\": container with ID starting with af5b80cc7b3d77e23e8b11c90e3725618fdf69d76ec9d0abc150ca5c9bd8469f not found: ID does not exist" containerID="af5b80cc7b3d77e23e8b11c90e3725618fdf69d76ec9d0abc150ca5c9bd8469f" Jul 29 01:27:18.031011 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:18.030968 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af5b80cc7b3d77e23e8b11c90e3725618fdf69d76ec9d0abc150ca5c9bd8469f"} err="failed to get container status \"af5b80cc7b3d77e23e8b11c90e3725618fdf69d76ec9d0abc150ca5c9bd8469f\": rpc error: code = NotFound desc = could not find container \"af5b80cc7b3d77e23e8b11c90e3725618fdf69d76ec9d0abc150ca5c9bd8469f\": container with ID starting with af5b80cc7b3d77e23e8b11c90e3725618fdf69d76ec9d0abc150ca5c9bd8469f not found: ID does not exist" Jul 29 01:27:18.031462 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:18.031443 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q"] Jul 29 01:27:18.037937 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:18.037914 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8c8da55e-openshift-default-68d79dff4-zgh2q"] Jul 29 01:27:18.343157 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:18.343126 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c6c2964-4fa6-4944-b083-b48e72f3c110" path="/var/lib/kubelet/pods/9c6c2964-4fa6-4944-b083-b48e72f3c110/volumes" Jul 29 01:27:29.066190 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:29.066158 2543 generic.go:356] "Generic (PLEG): container finished" podID="d779a0e7-7e7a-4219-ad72-3c48f1480cc6" containerID="8534f0095d89b01b8c20561f210d2a9bde433940bcf4b289ce9beab37591dd3b" exitCode=0 Jul 29 01:27:29.066700 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:29.066235 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" event={"ID":"d779a0e7-7e7a-4219-ad72-3c48f1480cc6","Type":"ContainerDied","Data":"8534f0095d89b01b8c20561f210d2a9bde433940bcf4b289ce9beab37591dd3b"} Jul 29 01:27:29.066770 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:29.066712 2543 scope.go:117] "RemoveContainer" containerID="8534f0095d89b01b8c20561f210d2a9bde433940bcf4b289ce9beab37591dd3b" Jul 29 01:27:29.067743 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:29.067715 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:27:29.067847 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:29.067778 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:27:29.067847 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:29.067808 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:27:29.992951 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:29.992905 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:30.072854 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:30.072748 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" event={"ID":"d779a0e7-7e7a-4219-ad72-3c48f1480cc6","Type":"ContainerStarted","Data":"f8b6a0cd552021dfc810e17b09263e38048e749b57d40c0657e0942381964cbf"} Jul 29 01:27:30.278671 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:30.278634 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:30.279614 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:30.279587 2543 patch_prober.go:28] interesting pod/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 29 01:27:30.279730 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:30.279641 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" podUID="d779a0e7-7e7a-4219-ad72-3c48f1480cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 29 01:27:31.278719 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:31.278676 2543 patch_prober.go:28] interesting pod/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 29 01:27:31.279152 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:31.278752 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" podUID="d779a0e7-7e7a-4219-ad72-3c48f1480cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 29 01:27:32.279218 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:32.279182 2543 patch_prober.go:28] interesting pod/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 29 01:27:32.279623 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:32.279251 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" podUID="d779a0e7-7e7a-4219-ad72-3c48f1480cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 29 01:27:33.279252 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:33.279209 2543 patch_prober.go:28] interesting pod/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 29 01:27:33.279646 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:33.279275 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" podUID="d779a0e7-7e7a-4219-ad72-3c48f1480cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 29 01:27:34.282838 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:34.282786 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:34.283224 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:34.283183 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:34.283941 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:34.283917 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:40.241461 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:40.241428 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c28408bd-nzdq5"] Jul 29 01:27:40.241857 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:40.241805 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c6c2964-4fa6-4944-b083-b48e72f3c110" containerName="istio-proxy" Jul 29 01:27:40.241857 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:40.241828 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c6c2964-4fa6-4944-b083-b48e72f3c110" containerName="istio-proxy" Jul 29 01:27:40.241941 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:40.241902 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c6c2964-4fa6-4944-b083-b48e72f3c110" containerName="istio-proxy" Jul 29 01:27:40.241941 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:40.241913 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c6c2964-4fa6-4944-b083-b48e72f3c110" containerName="istio-proxy" Jul 29 01:27:40.243910 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:40.243894 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c28408bd-nzdq5" Jul 29 01:27:40.246486 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:40.246465 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 01:27:40.246604 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:40.246526 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjvns\"" Jul 29 01:27:40.251855 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:40.251832 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c28408bd-nzdq5"] Jul 29 01:27:40.386240 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:40.386208 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05458b32-0cb3-49fc-a814-18b06940ddd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c28408bd-nzdq5\" (UID: \"05458b32-0cb3-49fc-a814-18b06940ddd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c28408bd-nzdq5" Jul 29 01:27:40.487600 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:40.487563 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05458b32-0cb3-49fc-a814-18b06940ddd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c28408bd-nzdq5\" (UID: \"05458b32-0cb3-49fc-a814-18b06940ddd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c28408bd-nzdq5" Jul 29 01:27:40.488183 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:40.488163 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05458b32-0cb3-49fc-a814-18b06940ddd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c28408bd-nzdq5\" (UID: \"05458b32-0cb3-49fc-a814-18b06940ddd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c28408bd-nzdq5" Jul 29 01:27:40.555466 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:40.555391 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c28408bd-nzdq5" Jul 29 01:27:40.683396 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:40.683371 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c28408bd-nzdq5"] Jul 29 01:27:40.685734 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:27:40.685704 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05458b32_0cb3_49fc_a814_18b06940ddd6.slice/crio-7ccfc7dcd0cad15181cd45c2486e5ab975284f12fed95d766e9cb7fde082711e WatchSource:0}: Error finding container 7ccfc7dcd0cad15181cd45c2486e5ab975284f12fed95d766e9cb7fde082711e: Status 404 returned error can't find the container with id 7ccfc7dcd0cad15181cd45c2486e5ab975284f12fed95d766e9cb7fde082711e Jul 29 01:27:41.128387 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:41.128354 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c28408bd-nzdq5" event={"ID":"05458b32-0cb3-49fc-a814-18b06940ddd6","Type":"ContainerStarted","Data":"f1fa2efc2ab59af25b96aeae21433b8012fbae6aa0b224f072244ecefebc0d15"} Jul 29 01:27:41.128666 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:41.128406 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c28408bd-nzdq5" event={"ID":"05458b32-0cb3-49fc-a814-18b06940ddd6","Type":"ContainerStarted","Data":"7ccfc7dcd0cad15181cd45c2486e5ab975284f12fed95d766e9cb7fde082711e"} Jul 29 01:27:41.143518 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:41.143469 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c28408bd-nzdq5" podStartSLOduration=1.1434542460000001 podStartE2EDuration="1.143454246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:27:41.141941554 +0000 UTC m=+2011.412585863" watchObservedRunningTime="2026-07-29 01:27:41.143454246 +0000 UTC m=+2011.414098553" Jul 29 01:27:42.134703 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:42.134668 2543 generic.go:356] "Generic (PLEG): container finished" podID="05458b32-0cb3-49fc-a814-18b06940ddd6" containerID="f1fa2efc2ab59af25b96aeae21433b8012fbae6aa0b224f072244ecefebc0d15" exitCode=0 Jul 29 01:27:42.135121 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:42.134751 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c28408bd-nzdq5" event={"ID":"05458b32-0cb3-49fc-a814-18b06940ddd6","Type":"ContainerDied","Data":"f1fa2efc2ab59af25b96aeae21433b8012fbae6aa0b224f072244ecefebc0d15"} Jul 29 01:27:43.269042 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:43.269016 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c28408bd-nzdq5" Jul 29 01:27:43.414450 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:43.414365 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05458b32-0cb3-49fc-a814-18b06940ddd6-maas-api-service-ca\") pod \"05458b32-0cb3-49fc-a814-18b06940ddd6\" (UID: \"05458b32-0cb3-49fc-a814-18b06940ddd6\") " Jul 29 01:27:43.414703 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:43.414683 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05458b32-0cb3-49fc-a814-18b06940ddd6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05458b32-0cb3-49fc-a814-18b06940ddd6" (UID: "05458b32-0cb3-49fc-a814-18b06940ddd6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:27:43.515902 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:43.515874 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05458b32-0cb3-49fc-a814-18b06940ddd6-maas-api-service-ca\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:27:44.145128 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:44.145096 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c28408bd-nzdq5" event={"ID":"05458b32-0cb3-49fc-a814-18b06940ddd6","Type":"ContainerDied","Data":"7ccfc7dcd0cad15181cd45c2486e5ab975284f12fed95d766e9cb7fde082711e"} Jul 29 01:27:44.145128 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:44.145126 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c28408bd-nzdq5" Jul 29 01:27:44.145335 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:44.145146 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ccfc7dcd0cad15181cd45c2486e5ab975284f12fed95d766e9cb7fde082711e" Jul 29 01:27:46.000188 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:46.000149 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c28408bd-nzdq5"] Jul 29 01:27:46.002031 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:46.002004 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c28408bd-nzdq5"] Jul 29 01:27:46.229227 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:46.229197 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb"] Jul 29 01:27:46.229449 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:46.229427 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" podUID="d779a0e7-7e7a-4219-ad72-3c48f1480cc6" containerName="istio-proxy" containerID="cri-o://f8b6a0cd552021dfc810e17b09263e38048e749b57d40c0657e0942381964cbf" gracePeriod=30 Jul 29 01:27:46.345101 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:46.345070 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05458b32-0cb3-49fc-a814-18b06940ddd6" path="/var/lib/kubelet/pods/05458b32-0cb3-49fc-a814-18b06940ddd6/volumes" Jul 29 01:27:49.285341 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:49.285303 2543 patch_prober.go:28] interesting pod/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 01:27:49.285731 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:49.285368 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" podUID="d779a0e7-7e7a-4219-ad72-3c48f1480cc6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 01:27:51.489866 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.489792 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:51.584686 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.584642 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-workload-certs\") pod \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " Jul 29 01:27:51.584686 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.584692 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-credential-socket\") pod \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " Jul 29 01:27:51.584969 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.584736 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-token\") pod \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " Jul 29 01:27:51.584969 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.584761 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-data\") pod \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " Jul 29 01:27:51.584969 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.584785 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-envoy\") pod \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " Jul 29 01:27:51.584969 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.584848 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istiod-ca-cert\") pod \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " Jul 29 01:27:51.584969 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.584880 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-workload-socket\") pod \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " Jul 29 01:27:51.584969 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.584900 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-podinfo\") pod \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " Jul 29 01:27:51.584969 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.584932 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n548s\" (UniqueName: \"kubernetes.io/projected/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-kube-api-access-n548s\") pod \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\" (UID: \"d779a0e7-7e7a-4219-ad72-3c48f1480cc6\") " Jul 29 01:27:51.585334 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.584983 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d779a0e7-7e7a-4219-ad72-3c48f1480cc6" (UID: "d779a0e7-7e7a-4219-ad72-3c48f1480cc6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:27:51.585334 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.585164 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d779a0e7-7e7a-4219-ad72-3c48f1480cc6" (UID: "d779a0e7-7e7a-4219-ad72-3c48f1480cc6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:27:51.585334 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.585263 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-workload-certs\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:27:51.585334 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.585265 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d779a0e7-7e7a-4219-ad72-3c48f1480cc6" (UID: "d779a0e7-7e7a-4219-ad72-3c48f1480cc6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:27:51.585334 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.585282 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-credential-socket\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:27:51.585581 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.585554 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d779a0e7-7e7a-4219-ad72-3c48f1480cc6" (UID: "d779a0e7-7e7a-4219-ad72-3c48f1480cc6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:27:51.585673 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.585643 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-data" (OuterVolumeSpecName: "istio-data") pod "d779a0e7-7e7a-4219-ad72-3c48f1480cc6" (UID: "d779a0e7-7e7a-4219-ad72-3c48f1480cc6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:27:51.587098 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.587074 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-token" (OuterVolumeSpecName: "istio-token") pod "d779a0e7-7e7a-4219-ad72-3c48f1480cc6" (UID: "d779a0e7-7e7a-4219-ad72-3c48f1480cc6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:27:51.587282 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.587251 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-kube-api-access-n548s" (OuterVolumeSpecName: "kube-api-access-n548s") pod "d779a0e7-7e7a-4219-ad72-3c48f1480cc6" (UID: "d779a0e7-7e7a-4219-ad72-3c48f1480cc6"). InnerVolumeSpecName "kube-api-access-n548s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:27:51.587381 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.587302 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d779a0e7-7e7a-4219-ad72-3c48f1480cc6" (UID: "d779a0e7-7e7a-4219-ad72-3c48f1480cc6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:27:51.587381 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.587304 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d779a0e7-7e7a-4219-ad72-3c48f1480cc6" (UID: "d779a0e7-7e7a-4219-ad72-3c48f1480cc6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 01:27:51.686425 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.686399 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-token\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:27:51.686425 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.686422 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-data\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:27:51.686425 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.686430 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-envoy\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:27:51.686666 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.686440 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istiod-ca-cert\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:27:51.686666 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.686448 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-workload-socket\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:27:51.686666 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.686456 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-istio-podinfo\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:27:51.686666 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:51.686464 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n548s\" (UniqueName: \"kubernetes.io/projected/d779a0e7-7e7a-4219-ad72-3c48f1480cc6-kube-api-access-n548s\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:27:52.188021 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:52.187986 2543 generic.go:356] "Generic (PLEG): container finished" podID="d779a0e7-7e7a-4219-ad72-3c48f1480cc6" containerID="f8b6a0cd552021dfc810e17b09263e38048e749b57d40c0657e0942381964cbf" exitCode=0 Jul 29 01:27:52.188193 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:52.188037 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" Jul 29 01:27:52.188193 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:52.188073 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" event={"ID":"d779a0e7-7e7a-4219-ad72-3c48f1480cc6","Type":"ContainerDied","Data":"f8b6a0cd552021dfc810e17b09263e38048e749b57d40c0657e0942381964cbf"} Jul 29 01:27:52.188193 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:52.188136 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb" event={"ID":"d779a0e7-7e7a-4219-ad72-3c48f1480cc6","Type":"ContainerDied","Data":"beea81e567e636dfb78c2d0108556c322d043d6a3748e07f185a5b210bf87493"} Jul 29 01:27:52.188193 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:52.188167 2543 scope.go:117] "RemoveContainer" containerID="f8b6a0cd552021dfc810e17b09263e38048e749b57d40c0657e0942381964cbf" Jul 29 01:27:52.197053 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:52.197035 2543 scope.go:117] "RemoveContainer" containerID="8534f0095d89b01b8c20561f210d2a9bde433940bcf4b289ce9beab37591dd3b" Jul 29 01:27:52.204511 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:52.204495 2543 scope.go:117] "RemoveContainer" containerID="f8b6a0cd552021dfc810e17b09263e38048e749b57d40c0657e0942381964cbf" Jul 29 01:27:52.204775 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:27:52.204756 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b6a0cd552021dfc810e17b09263e38048e749b57d40c0657e0942381964cbf\": container with ID starting with f8b6a0cd552021dfc810e17b09263e38048e749b57d40c0657e0942381964cbf not found: ID does not exist" containerID="f8b6a0cd552021dfc810e17b09263e38048e749b57d40c0657e0942381964cbf" Jul 29 01:27:52.204838 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:52.204782 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b6a0cd552021dfc810e17b09263e38048e749b57d40c0657e0942381964cbf"} err="failed to get container status \"f8b6a0cd552021dfc810e17b09263e38048e749b57d40c0657e0942381964cbf\": rpc error: code = NotFound desc = could not find container \"f8b6a0cd552021dfc810e17b09263e38048e749b57d40c0657e0942381964cbf\": container with ID starting with f8b6a0cd552021dfc810e17b09263e38048e749b57d40c0657e0942381964cbf not found: ID does not exist" Jul 29 01:27:52.204838 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:52.204798 2543 scope.go:117] "RemoveContainer" containerID="8534f0095d89b01b8c20561f210d2a9bde433940bcf4b289ce9beab37591dd3b" Jul 29 01:27:52.205037 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:27:52.205019 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8534f0095d89b01b8c20561f210d2a9bde433940bcf4b289ce9beab37591dd3b\": container with ID starting with 8534f0095d89b01b8c20561f210d2a9bde433940bcf4b289ce9beab37591dd3b not found: ID does not exist" containerID="8534f0095d89b01b8c20561f210d2a9bde433940bcf4b289ce9beab37591dd3b" Jul 29 01:27:52.205083 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:52.205044 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8534f0095d89b01b8c20561f210d2a9bde433940bcf4b289ce9beab37591dd3b"} err="failed to get container status \"8534f0095d89b01b8c20561f210d2a9bde433940bcf4b289ce9beab37591dd3b\": rpc error: code = NotFound desc = could not find container \"8534f0095d89b01b8c20561f210d2a9bde433940bcf4b289ce9beab37591dd3b\": container with ID starting with 8534f0095d89b01b8c20561f210d2a9bde433940bcf4b289ce9beab37591dd3b not found: ID does not exist" Jul 29 01:27:52.210014 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:52.209987 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb"] Jul 29 01:27:52.214617 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:52.214598 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c28408bd-openshift-default-7758fd96d8-2lksb"] Jul 29 01:27:52.342783 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:27:52.342751 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d779a0e7-7e7a-4219-ad72-3c48f1480cc6" path="/var/lib/kubelet/pods/d779a0e7-7e7a-4219-ad72-3c48f1480cc6/volumes" Jul 29 01:29:10.662754 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:10.662724 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 01:29:10.991789 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:10.991709 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 01:29:36.669548 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.669514 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls"] Jul 29 01:29:36.670199 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.670005 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c6c2964-4fa6-4944-b083-b48e72f3c110" containerName="istio-proxy" Jul 29 01:29:36.670199 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.670025 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c6c2964-4fa6-4944-b083-b48e72f3c110" containerName="istio-proxy" Jul 29 01:29:36.670199 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.670039 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d779a0e7-7e7a-4219-ad72-3c48f1480cc6" containerName="istio-proxy" Jul 29 01:29:36.670199 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.670048 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d779a0e7-7e7a-4219-ad72-3c48f1480cc6" containerName="istio-proxy" Jul 29 01:29:36.670199 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.670059 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d779a0e7-7e7a-4219-ad72-3c48f1480cc6" containerName="istio-proxy" Jul 29 01:29:36.670199 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.670068 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d779a0e7-7e7a-4219-ad72-3c48f1480cc6" containerName="istio-proxy" Jul 29 01:29:36.670199 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.670099 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05458b32-0cb3-49fc-a814-18b06940ddd6" containerName="revoke-keys" Jul 29 01:29:36.670199 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.670108 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="05458b32-0cb3-49fc-a814-18b06940ddd6" containerName="revoke-keys" Jul 29 01:29:36.670652 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.670209 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="05458b32-0cb3-49fc-a814-18b06940ddd6" containerName="revoke-keys" Jul 29 01:29:36.670652 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.670222 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="d779a0e7-7e7a-4219-ad72-3c48f1480cc6" containerName="istio-proxy" Jul 29 01:29:36.670652 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.670233 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="d779a0e7-7e7a-4219-ad72-3c48f1480cc6" containerName="istio-proxy" Jul 29 01:29:36.673259 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.673240 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.675564 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.675544 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4bfdf293-openshift-default-dockercfg-6jlfl\"" Jul 29 01:29:36.695362 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.695335 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls"] Jul 29 01:29:36.762671 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.762633 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwcb4\" (UniqueName: \"kubernetes.io/projected/9a710f3d-e11a-455b-9d98-378f1595b54a-kube-api-access-dwcb4\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.762872 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.762692 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-token\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.762872 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.762745 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a710f3d-e11a-455b-9d98-378f1595b54a-istiod-ca-cert\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.762872 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.762766 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-workload-socket\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.762872 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.762783 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-credential-socket\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.762872 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.762799 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-data\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.762872 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.762835 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-podinfo\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.763220 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.762916 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-workload-certs\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.763220 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.762978 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-envoy\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.864065 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.864030 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwcb4\" (UniqueName: \"kubernetes.io/projected/9a710f3d-e11a-455b-9d98-378f1595b54a-kube-api-access-dwcb4\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.864262 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.864096 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-token\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.864262 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.864136 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a710f3d-e11a-455b-9d98-378f1595b54a-istiod-ca-cert\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.864262 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.864161 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-workload-socket\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.864417 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.864262 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-credential-socket\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.864417 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.864285 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-data\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.864417 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.864311 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-podinfo\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.864417 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.864353 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-workload-certs\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.864417 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.864409 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-envoy\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.864659 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.864572 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-workload-socket\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.864713 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.864673 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-credential-socket\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.864775 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.864746 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-data\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.864854 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.864755 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-workload-certs\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.865005 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.864983 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a710f3d-e11a-455b-9d98-378f1595b54a-istiod-ca-cert\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.866567 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.866543 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-envoy\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.866786 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.866768 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-podinfo\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.879197 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.879178 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-token\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.881118 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.881092 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwcb4\" (UniqueName: \"kubernetes.io/projected/9a710f3d-e11a-455b-9d98-378f1595b54a-kube-api-access-dwcb4\") pod \"e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:36.983221 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:36.983134 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:37.116854 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:37.116828 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls"] Jul 29 01:29:37.118805 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:29:37.118779 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a710f3d_e11a_455b_9d98_378f1595b54a.slice/crio-e67171f507acd520cf02e6a2e1b67a36ae2cfa7c745044dcf411e478ecb5c3cc WatchSource:0}: Error finding container e67171f507acd520cf02e6a2e1b67a36ae2cfa7c745044dcf411e478ecb5c3cc: Status 404 returned error can't find the container with id e67171f507acd520cf02e6a2e1b67a36ae2cfa7c745044dcf411e478ecb5c3cc Jul 29 01:29:37.120935 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:37.120903 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:29:37.121027 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:37.120964 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:29:37.121027 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:37.120991 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:29:37.700728 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:37.700686 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" event={"ID":"9a710f3d-e11a-455b-9d98-378f1595b54a","Type":"ContainerStarted","Data":"7c52eef2418c72480a29ff01286c0ce542483cf4fd649655cc94b7f896187244"} Jul 29 01:29:37.701135 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:37.700777 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" event={"ID":"9a710f3d-e11a-455b-9d98-378f1595b54a","Type":"ContainerStarted","Data":"e67171f507acd520cf02e6a2e1b67a36ae2cfa7c745044dcf411e478ecb5c3cc"} Jul 29 01:29:37.720828 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:37.720781 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" podStartSLOduration=1.720764993 podStartE2EDuration="1.720764993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:29:37.719455113 +0000 UTC m=+2127.990099433" watchObservedRunningTime="2026-07-29 01:29:37.720764993 +0000 UTC m=+2127.991409300" Jul 29 01:29:37.983644 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:37.983567 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:38.989181 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:38.989150 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:39.710411 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:39.710383 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:39.711363 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:39.711344 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:53.779514 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:53.779480 2543 generic.go:356] "Generic (PLEG): container finished" podID="9a710f3d-e11a-455b-9d98-378f1595b54a" containerID="7c52eef2418c72480a29ff01286c0ce542483cf4fd649655cc94b7f896187244" exitCode=0 Jul 29 01:29:53.779966 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:53.779551 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" event={"ID":"9a710f3d-e11a-455b-9d98-378f1595b54a","Type":"ContainerDied","Data":"7c52eef2418c72480a29ff01286c0ce542483cf4fd649655cc94b7f896187244"} Jul 29 01:29:53.779966 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:53.779954 2543 scope.go:117] "RemoveContainer" containerID="7c52eef2418c72480a29ff01286c0ce542483cf4fd649655cc94b7f896187244" Jul 29 01:29:53.780965 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:53.780939 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:29:53.781035 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:53.781010 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:29:53.781071 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:53.781049 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:29:54.710416 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:54.710361 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:54.786015 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:54.785979 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" event={"ID":"9a710f3d-e11a-455b-9d98-378f1595b54a","Type":"ContainerStarted","Data":"6a4a474023ee0ca5202e12caaacebca168d2382dc9e5ec1fac333f7f11d19ef9"} Jul 29 01:29:54.983890 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:54.983762 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:54.984930 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:54.984894 2543 patch_prober.go:28] interesting pod/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 29 01:29:54.985061 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:54.984983 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" podUID="9a710f3d-e11a-455b-9d98-378f1595b54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 29 01:29:55.984420 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:55.984381 2543 patch_prober.go:28] interesting pod/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 29 01:29:55.984940 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:55.984460 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" podUID="9a710f3d-e11a-455b-9d98-378f1595b54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 29 01:29:56.983648 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:56.983614 2543 patch_prober.go:28] interesting pod/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 29 01:29:56.983852 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:56.983674 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" podUID="9a710f3d-e11a-455b-9d98-378f1595b54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 29 01:29:57.984529 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:57.984493 2543 patch_prober.go:28] interesting pod/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 29 01:29:57.984943 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:57.984550 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" podUID="9a710f3d-e11a-455b-9d98-378f1595b54a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 29 01:29:58.988554 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:58.988520 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:58.988952 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:58.988851 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:29:58.989633 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:29:58.989611 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:30:00.134248 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:00.134206 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754810-7875b"] Jul 29 01:30:00.137118 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:00.137096 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4bfdf293-29754810-pmnq7"] Jul 29 01:30:00.137238 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:00.137222 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754810-7875b" Jul 29 01:30:00.139394 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:00.139372 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjvns\"" Jul 29 01:30:00.139533 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:00.139515 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4bfdf293-29754810-pmnq7" Jul 29 01:30:00.146013 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:00.145418 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754810-7875b"] Jul 29 01:30:00.148419 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:00.148397 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4bfdf293-29754810-pmnq7"] Jul 29 01:30:00.151067 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:00.151037 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754810-7875b" Jul 29 01:30:00.163929 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:00.163902 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4bfdf293-29754810-pmnq7" Jul 29 01:30:00.307681 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:00.307646 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754810-7875b"] Jul 29 01:30:00.309402 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:30:00.309337 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34540ad_bcb5_4ec5_9f44_e203bb03415e.slice/crio-f279c2d69985c2502c34f5575fd5989d44b96f2d8a43b3d983b923937d2fb623 WatchSource:0}: Error finding container f279c2d69985c2502c34f5575fd5989d44b96f2d8a43b3d983b923937d2fb623: Status 404 returned error can't find the container with id f279c2d69985c2502c34f5575fd5989d44b96f2d8a43b3d983b923937d2fb623 Jul 29 01:30:00.332488 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:00.332458 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4bfdf293-29754810-pmnq7"] Jul 29 01:30:00.335835 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:30:00.335795 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2bf122_691d_496e_badf_28f94584e1df.slice/crio-c5bbc1b05ecc14e35f7af2b3dc2c45c05a123d6f885d4d9ef77646f2f6f40d97 WatchSource:0}: Error finding container c5bbc1b05ecc14e35f7af2b3dc2c45c05a123d6f885d4d9ef77646f2f6f40d97: Status 404 returned error can't find the container with id c5bbc1b05ecc14e35f7af2b3dc2c45c05a123d6f885d4d9ef77646f2f6f40d97 Jul 29 01:30:00.824307 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:00.824202 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4bfdf293-29754810-pmnq7" event={"ID":"df2bf122-691d-496e-badf-28f94584e1df","Type":"ContainerStarted","Data":"d650c51bb35110251fcd7d5bde959e3d475c69e6698ef463e6a06555f1f08fcb"} Jul 29 01:30:00.824307 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:00.824269 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4bfdf293-29754810-pmnq7" event={"ID":"df2bf122-691d-496e-badf-28f94584e1df","Type":"ContainerStarted","Data":"c5bbc1b05ecc14e35f7af2b3dc2c45c05a123d6f885d4d9ef77646f2f6f40d97"} Jul 29 01:30:00.825650 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:00.825618 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754810-7875b" event={"ID":"f34540ad-bcb5-4ec5-9f44-e203bb03415e","Type":"ContainerStarted","Data":"2e5417fbe4b8c999c5542e4a46ba70849d6d8e8f7d03c9a6dd24f6e0eb906dfd"} Jul 29 01:30:00.825775 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:00.825673 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754810-7875b" event={"ID":"f34540ad-bcb5-4ec5-9f44-e203bb03415e","Type":"ContainerStarted","Data":"f279c2d69985c2502c34f5575fd5989d44b96f2d8a43b3d983b923937d2fb623"} Jul 29 01:30:00.838134 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:00.838088 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4bfdf293-29754810-pmnq7" podStartSLOduration=0.838072975 podStartE2EDuration="838.072975ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:30:00.83716124 +0000 UTC m=+2151.107805547" watchObservedRunningTime="2026-07-29 01:30:00.838072975 +0000 UTC m=+2151.108717281" Jul 29 01:30:00.851328 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:00.851283 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754810-7875b" podStartSLOduration=0.851269006 podStartE2EDuration="851.269006ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:30:00.848975882 +0000 UTC m=+2151.119620207" watchObservedRunningTime="2026-07-29 01:30:00.851269006 +0000 UTC m=+2151.121913312" Jul 29 01:30:01.832777 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:01.832743 2543 generic.go:356] "Generic (PLEG): container finished" podID="df2bf122-691d-496e-badf-28f94584e1df" containerID="d650c51bb35110251fcd7d5bde959e3d475c69e6698ef463e6a06555f1f08fcb" exitCode=0 Jul 29 01:30:01.833230 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:01.832792 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4bfdf293-29754810-pmnq7" event={"ID":"df2bf122-691d-496e-badf-28f94584e1df","Type":"ContainerDied","Data":"d650c51bb35110251fcd7d5bde959e3d475c69e6698ef463e6a06555f1f08fcb"} Jul 29 01:30:01.834335 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:01.834307 2543 generic.go:356] "Generic (PLEG): container finished" podID="f34540ad-bcb5-4ec5-9f44-e203bb03415e" containerID="2e5417fbe4b8c999c5542e4a46ba70849d6d8e8f7d03c9a6dd24f6e0eb906dfd" exitCode=0 Jul 29 01:30:01.834458 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:01.834391 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754810-7875b" event={"ID":"f34540ad-bcb5-4ec5-9f44-e203bb03415e","Type":"ContainerDied","Data":"2e5417fbe4b8c999c5542e4a46ba70849d6d8e8f7d03c9a6dd24f6e0eb906dfd"} Jul 29 01:30:02.975984 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:02.975959 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754810-7875b" Jul 29 01:30:02.999517 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:02.999488 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4bfdf293-29754810-pmnq7" Jul 29 01:30:03.845931 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:03.845903 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4bfdf293-29754810-pmnq7" Jul 29 01:30:03.846120 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:03.845905 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4bfdf293-29754810-pmnq7" event={"ID":"df2bf122-691d-496e-badf-28f94584e1df","Type":"ContainerDied","Data":"c5bbc1b05ecc14e35f7af2b3dc2c45c05a123d6f885d4d9ef77646f2f6f40d97"} Jul 29 01:30:03.846120 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:03.846025 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5bbc1b05ecc14e35f7af2b3dc2c45c05a123d6f885d4d9ef77646f2f6f40d97" Jul 29 01:30:03.847083 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:03.847065 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754810-7875b" Jul 29 01:30:03.847198 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:03.847103 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754810-7875b" event={"ID":"f34540ad-bcb5-4ec5-9f44-e203bb03415e","Type":"ContainerDied","Data":"f279c2d69985c2502c34f5575fd5989d44b96f2d8a43b3d983b923937d2fb623"} Jul 29 01:30:03.847198 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:03.847132 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f279c2d69985c2502c34f5575fd5989d44b96f2d8a43b3d983b923937d2fb623" Jul 29 01:30:11.822050 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:11.822013 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4bfdf293-hllmm"] Jul 29 01:30:11.822507 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:11.822399 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df2bf122-691d-496e-badf-28f94584e1df" containerName="cleanup" Jul 29 01:30:11.822507 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:11.822414 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="df2bf122-691d-496e-badf-28f94584e1df" containerName="cleanup" Jul 29 01:30:11.822507 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:11.822430 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f34540ad-bcb5-4ec5-9f44-e203bb03415e" containerName="cleanup" Jul 29 01:30:11.822507 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:11.822435 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f34540ad-bcb5-4ec5-9f44-e203bb03415e" containerName="cleanup" Jul 29 01:30:11.822507 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:11.822496 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="df2bf122-691d-496e-badf-28f94584e1df" containerName="cleanup" Jul 29 01:30:11.822507 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:11.822510 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="f34540ad-bcb5-4ec5-9f44-e203bb03415e" containerName="cleanup" Jul 29 01:30:11.825625 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:11.825609 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4bfdf293-hllmm" Jul 29 01:30:11.827724 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:11.827700 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 01:30:11.827871 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:11.827744 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjvns\"" Jul 29 01:30:11.832225 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:11.832202 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4bfdf293-hllmm"] Jul 29 01:30:11.875367 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:11.875338 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f369ec-96e3-49eb-b06e-16c59249441a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4bfdf293-hllmm\" (UID: \"c4f369ec-96e3-49eb-b06e-16c59249441a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4bfdf293-hllmm" Jul 29 01:30:11.975984 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:11.975946 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f369ec-96e3-49eb-b06e-16c59249441a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4bfdf293-hllmm\" (UID: \"c4f369ec-96e3-49eb-b06e-16c59249441a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4bfdf293-hllmm" Jul 29 01:30:11.976559 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:11.976539 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f369ec-96e3-49eb-b06e-16c59249441a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4bfdf293-hllmm\" (UID: \"c4f369ec-96e3-49eb-b06e-16c59249441a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4bfdf293-hllmm" Jul 29 01:30:12.138405 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:12.138378 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4bfdf293-hllmm" Jul 29 01:30:12.265653 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:12.265630 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4bfdf293-hllmm"] Jul 29 01:30:12.267514 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:30:12.267489 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f369ec_96e3_49eb_b06e_16c59249441a.slice/crio-1a84d842b415353f530168ef8cf86b07cb26306538fb2bd1360a288e64c0e24a WatchSource:0}: Error finding container 1a84d842b415353f530168ef8cf86b07cb26306538fb2bd1360a288e64c0e24a: Status 404 returned error can't find the container with id 1a84d842b415353f530168ef8cf86b07cb26306538fb2bd1360a288e64c0e24a Jul 29 01:30:12.899668 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:12.899631 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4bfdf293-hllmm" event={"ID":"c4f369ec-96e3-49eb-b06e-16c59249441a","Type":"ContainerStarted","Data":"5c79818a51d1d241e6ccc7147326b40790b7904d999b66bd306f4d3edc916de9"} Jul 29 01:30:12.900115 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:12.899692 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4bfdf293-hllmm" event={"ID":"c4f369ec-96e3-49eb-b06e-16c59249441a","Type":"ContainerStarted","Data":"1a84d842b415353f530168ef8cf86b07cb26306538fb2bd1360a288e64c0e24a"} Jul 29 01:30:12.914261 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:12.914217 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4bfdf293-hllmm" podStartSLOduration=1.914202016 podStartE2EDuration="1.914202016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:30:12.913032274 +0000 UTC m=+2163.183676598" watchObservedRunningTime="2026-07-29 01:30:12.914202016 +0000 UTC m=+2163.184846322" Jul 29 01:30:13.905765 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:13.905731 2543 generic.go:356] "Generic (PLEG): container finished" podID="c4f369ec-96e3-49eb-b06e-16c59249441a" containerID="5c79818a51d1d241e6ccc7147326b40790b7904d999b66bd306f4d3edc916de9" exitCode=0 Jul 29 01:30:13.906221 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:13.905778 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4bfdf293-hllmm" event={"ID":"c4f369ec-96e3-49eb-b06e-16c59249441a","Type":"ContainerDied","Data":"5c79818a51d1d241e6ccc7147326b40790b7904d999b66bd306f4d3edc916de9"} Jul 29 01:30:15.039449 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:15.039422 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4bfdf293-hllmm" Jul 29 01:30:15.105050 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:15.105006 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f369ec-96e3-49eb-b06e-16c59249441a-maas-api-service-ca\") pod \"c4f369ec-96e3-49eb-b06e-16c59249441a\" (UID: \"c4f369ec-96e3-49eb-b06e-16c59249441a\") " Jul 29 01:30:15.105351 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:15.105328 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f369ec-96e3-49eb-b06e-16c59249441a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c4f369ec-96e3-49eb-b06e-16c59249441a" (UID: "c4f369ec-96e3-49eb-b06e-16c59249441a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:30:15.206460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:15.206380 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4f369ec-96e3-49eb-b06e-16c59249441a-maas-api-service-ca\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:30:15.916554 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:15.916521 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4bfdf293-hllmm" Jul 29 01:30:15.916747 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:15.916566 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4bfdf293-hllmm" event={"ID":"c4f369ec-96e3-49eb-b06e-16c59249441a","Type":"ContainerDied","Data":"1a84d842b415353f530168ef8cf86b07cb26306538fb2bd1360a288e64c0e24a"} Jul 29 01:30:15.916747 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:15.916654 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a84d842b415353f530168ef8cf86b07cb26306538fb2bd1360a288e64c0e24a" Jul 29 01:30:22.366132 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:22.366098 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4bfdf293-29754810-pmnq7"] Jul 29 01:30:22.371448 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:22.371422 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-4bfdf293-29754810-pmnq7"] Jul 29 01:30:22.560369 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:22.560338 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4bfdf293-hllmm"] Jul 29 01:30:22.563978 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:22.563948 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4bfdf293-hllmm"] Jul 29 01:30:24.343489 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:24.343461 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4f369ec-96e3-49eb-b06e-16c59249441a" path="/var/lib/kubelet/pods/c4f369ec-96e3-49eb-b06e-16c59249441a/volumes" Jul 29 01:30:24.343958 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:24.343759 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df2bf122-691d-496e-badf-28f94584e1df" path="/var/lib/kubelet/pods/df2bf122-691d-496e-badf-28f94584e1df/volumes" Jul 29 01:30:28.994057 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:28.994023 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls"] Jul 29 01:30:28.994519 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:28.994261 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" podUID="9a710f3d-e11a-455b-9d98-378f1595b54a" containerName="istio-proxy" containerID="cri-o://6a4a474023ee0ca5202e12caaacebca168d2382dc9e5ec1fac333f7f11d19ef9" gracePeriod=30 Jul 29 01:30:34.243638 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.243613 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:30:34.380744 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.380707 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-data\") pod \"9a710f3d-e11a-455b-9d98-378f1595b54a\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " Jul 29 01:30:34.380971 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.380750 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-podinfo\") pod \"9a710f3d-e11a-455b-9d98-378f1595b54a\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " Jul 29 01:30:34.380971 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.380775 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-credential-socket\") pod \"9a710f3d-e11a-455b-9d98-378f1595b54a\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " Jul 29 01:30:34.380971 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.380802 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-token\") pod \"9a710f3d-e11a-455b-9d98-378f1595b54a\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " Jul 29 01:30:34.380971 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.380842 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-workload-socket\") pod \"9a710f3d-e11a-455b-9d98-378f1595b54a\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " Jul 29 01:30:34.380971 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.380880 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-envoy\") pod \"9a710f3d-e11a-455b-9d98-378f1595b54a\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " Jul 29 01:30:34.380971 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.380922 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a710f3d-e11a-455b-9d98-378f1595b54a-istiod-ca-cert\") pod \"9a710f3d-e11a-455b-9d98-378f1595b54a\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " Jul 29 01:30:34.381269 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.380993 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwcb4\" (UniqueName: \"kubernetes.io/projected/9a710f3d-e11a-455b-9d98-378f1595b54a-kube-api-access-dwcb4\") pod \"9a710f3d-e11a-455b-9d98-378f1595b54a\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " Jul 29 01:30:34.381269 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.381021 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-workload-certs\") pod \"9a710f3d-e11a-455b-9d98-378f1595b54a\" (UID: \"9a710f3d-e11a-455b-9d98-378f1595b54a\") " Jul 29 01:30:34.381269 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.381191 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9a710f3d-e11a-455b-9d98-378f1595b54a" (UID: "9a710f3d-e11a-455b-9d98-378f1595b54a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:30:34.381424 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.381322 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-workload-socket\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:30:34.381424 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.381321 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a710f3d-e11a-455b-9d98-378f1595b54a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9a710f3d-e11a-455b-9d98-378f1595b54a" (UID: "9a710f3d-e11a-455b-9d98-378f1595b54a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:30:34.381424 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.381393 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9a710f3d-e11a-455b-9d98-378f1595b54a" (UID: "9a710f3d-e11a-455b-9d98-378f1595b54a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:30:34.381560 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.381449 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9a710f3d-e11a-455b-9d98-378f1595b54a" (UID: "9a710f3d-e11a-455b-9d98-378f1595b54a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:30:34.381860 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.381834 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-data" (OuterVolumeSpecName: "istio-data") pod "9a710f3d-e11a-455b-9d98-378f1595b54a" (UID: "9a710f3d-e11a-455b-9d98-378f1595b54a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:30:34.383100 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.383071 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9a710f3d-e11a-455b-9d98-378f1595b54a" (UID: "9a710f3d-e11a-455b-9d98-378f1595b54a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:30:34.383200 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.383146 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-token" (OuterVolumeSpecName: "istio-token") pod "9a710f3d-e11a-455b-9d98-378f1595b54a" (UID: "9a710f3d-e11a-455b-9d98-378f1595b54a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:30:34.383387 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.383372 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9a710f3d-e11a-455b-9d98-378f1595b54a" (UID: "9a710f3d-e11a-455b-9d98-378f1595b54a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 01:30:34.383516 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.383495 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a710f3d-e11a-455b-9d98-378f1595b54a-kube-api-access-dwcb4" (OuterVolumeSpecName: "kube-api-access-dwcb4") pod "9a710f3d-e11a-455b-9d98-378f1595b54a" (UID: "9a710f3d-e11a-455b-9d98-378f1595b54a"). InnerVolumeSpecName "kube-api-access-dwcb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:30:34.482055 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.482019 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-data\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:30:34.482055 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.482047 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-podinfo\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:30:34.482055 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.482057 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-credential-socket\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:30:34.482055 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.482067 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-token\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:30:34.482317 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.482075 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-istio-envoy\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:30:34.482317 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.482084 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a710f3d-e11a-455b-9d98-378f1595b54a-istiod-ca-cert\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:30:34.482317 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.482092 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwcb4\" (UniqueName: \"kubernetes.io/projected/9a710f3d-e11a-455b-9d98-378f1595b54a-kube-api-access-dwcb4\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:30:34.482317 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:34.482101 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a710f3d-e11a-455b-9d98-378f1595b54a-workload-certs\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:30:35.010920 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:35.010875 2543 generic.go:356] "Generic (PLEG): container finished" podID="9a710f3d-e11a-455b-9d98-378f1595b54a" containerID="6a4a474023ee0ca5202e12caaacebca168d2382dc9e5ec1fac333f7f11d19ef9" exitCode=0 Jul 29 01:30:35.011187 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:35.010937 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" Jul 29 01:30:35.011187 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:35.010958 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" event={"ID":"9a710f3d-e11a-455b-9d98-378f1595b54a","Type":"ContainerDied","Data":"6a4a474023ee0ca5202e12caaacebca168d2382dc9e5ec1fac333f7f11d19ef9"} Jul 29 01:30:35.011187 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:35.011043 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls" event={"ID":"9a710f3d-e11a-455b-9d98-378f1595b54a","Type":"ContainerDied","Data":"e67171f507acd520cf02e6a2e1b67a36ae2cfa7c745044dcf411e478ecb5c3cc"} Jul 29 01:30:35.011187 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:35.011073 2543 scope.go:117] "RemoveContainer" containerID="6a4a474023ee0ca5202e12caaacebca168d2382dc9e5ec1fac333f7f11d19ef9" Jul 29 01:30:35.020186 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:35.020168 2543 scope.go:117] "RemoveContainer" containerID="7c52eef2418c72480a29ff01286c0ce542483cf4fd649655cc94b7f896187244" Jul 29 01:30:35.028209 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:35.028194 2543 scope.go:117] "RemoveContainer" containerID="6a4a474023ee0ca5202e12caaacebca168d2382dc9e5ec1fac333f7f11d19ef9" Jul 29 01:30:35.028458 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:30:35.028438 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a4a474023ee0ca5202e12caaacebca168d2382dc9e5ec1fac333f7f11d19ef9\": container with ID starting with 6a4a474023ee0ca5202e12caaacebca168d2382dc9e5ec1fac333f7f11d19ef9 not found: ID does not exist" containerID="6a4a474023ee0ca5202e12caaacebca168d2382dc9e5ec1fac333f7f11d19ef9" Jul 29 01:30:35.028533 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:35.028465 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a4a474023ee0ca5202e12caaacebca168d2382dc9e5ec1fac333f7f11d19ef9"} err="failed to get container status \"6a4a474023ee0ca5202e12caaacebca168d2382dc9e5ec1fac333f7f11d19ef9\": rpc error: code = NotFound desc = could not find container \"6a4a474023ee0ca5202e12caaacebca168d2382dc9e5ec1fac333f7f11d19ef9\": container with ID starting with 6a4a474023ee0ca5202e12caaacebca168d2382dc9e5ec1fac333f7f11d19ef9 not found: ID does not exist" Jul 29 01:30:35.028533 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:35.028483 2543 scope.go:117] "RemoveContainer" containerID="7c52eef2418c72480a29ff01286c0ce542483cf4fd649655cc94b7f896187244" Jul 29 01:30:35.028790 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:30:35.028765 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c52eef2418c72480a29ff01286c0ce542483cf4fd649655cc94b7f896187244\": container with ID starting with 7c52eef2418c72480a29ff01286c0ce542483cf4fd649655cc94b7f896187244 not found: ID does not exist" containerID="7c52eef2418c72480a29ff01286c0ce542483cf4fd649655cc94b7f896187244" Jul 29 01:30:35.028878 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:35.028799 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c52eef2418c72480a29ff01286c0ce542483cf4fd649655cc94b7f896187244"} err="failed to get container status \"7c52eef2418c72480a29ff01286c0ce542483cf4fd649655cc94b7f896187244\": rpc error: code = NotFound desc = could not find container \"7c52eef2418c72480a29ff01286c0ce542483cf4fd649655cc94b7f896187244\": container with ID starting with 7c52eef2418c72480a29ff01286c0ce542483cf4fd649655cc94b7f896187244 not found: ID does not exist" Jul 29 01:30:35.031306 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:35.031285 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls"] Jul 29 01:30:35.035154 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:35.035136 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4bfdf293-openshift-default-7c799844d4-569ls"] Jul 29 01:30:36.342941 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:36.342908 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a710f3d-e11a-455b-9d98-378f1595b54a" path="/var/lib/kubelet/pods/9a710f3d-e11a-455b-9d98-378f1595b54a/volumes" Jul 29 01:30:48.328490 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.328459 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk"] Jul 29 01:30:48.328875 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.328796 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f369ec-96e3-49eb-b06e-16c59249441a" containerName="revoke-keys" Jul 29 01:30:48.328875 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.328808 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f369ec-96e3-49eb-b06e-16c59249441a" containerName="revoke-keys" Jul 29 01:30:48.328875 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.328843 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a710f3d-e11a-455b-9d98-378f1595b54a" containerName="istio-proxy" Jul 29 01:30:48.328875 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.328850 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a710f3d-e11a-455b-9d98-378f1595b54a" containerName="istio-proxy" Jul 29 01:30:48.328875 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.328858 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a710f3d-e11a-455b-9d98-378f1595b54a" containerName="istio-proxy" Jul 29 01:30:48.328875 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.328863 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a710f3d-e11a-455b-9d98-378f1595b54a" containerName="istio-proxy" Jul 29 01:30:48.329075 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.328927 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a710f3d-e11a-455b-9d98-378f1595b54a" containerName="istio-proxy" Jul 29 01:30:48.329075 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.328934 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a710f3d-e11a-455b-9d98-378f1595b54a" containerName="istio-proxy" Jul 29 01:30:48.329075 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.328939 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4f369ec-96e3-49eb-b06e-16c59249441a" containerName="revoke-keys" Jul 29 01:30:48.331755 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.331737 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.333852 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.333829 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f46b2363-openshift-default-dockercfg-gn85v\"" Jul 29 01:30:48.346396 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.346374 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk"] Jul 29 01:30:48.384702 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.384667 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-workload-certs\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.384702 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.384699 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-istio-envoy\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.384933 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.384724 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-workload-socket\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.384933 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.384743 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-istio-data\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.384933 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.384784 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f05af9-1c21-479d-8db7-9e3085f84184-istio-podinfo\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.384933 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.384807 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f05af9-1c21-479d-8db7-9e3085f84184-istiod-ca-cert\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.384933 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.384857 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f05af9-1c21-479d-8db7-9e3085f84184-istio-token\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.384933 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.384873 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdjvb\" (UniqueName: \"kubernetes.io/projected/57f05af9-1c21-479d-8db7-9e3085f84184-kube-api-access-fdjvb\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.385127 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.384963 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-credential-socket\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.485872 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.485835 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f05af9-1c21-479d-8db7-9e3085f84184-istiod-ca-cert\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.485872 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.485874 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f05af9-1c21-479d-8db7-9e3085f84184-istio-token\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.486117 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.485892 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdjvb\" (UniqueName: \"kubernetes.io/projected/57f05af9-1c21-479d-8db7-9e3085f84184-kube-api-access-fdjvb\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.486117 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.485930 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-credential-socket\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.486117 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.485973 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-workload-certs\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.486117 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.485988 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-istio-envoy\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.486117 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.486010 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-workload-socket\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.486117 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.486035 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-istio-data\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.486117 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.486077 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f05af9-1c21-479d-8db7-9e3085f84184-istio-podinfo\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.486489 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.486394 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-credential-socket\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.486489 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.486423 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-workload-certs\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.486489 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.486466 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-istio-data\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.486592 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.486493 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-workload-socket\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.486592 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.486573 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f05af9-1c21-479d-8db7-9e3085f84184-istiod-ca-cert\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.488319 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.488293 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-istio-envoy\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.488516 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.488500 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f05af9-1c21-479d-8db7-9e3085f84184-istio-podinfo\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.502877 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.502858 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f05af9-1c21-479d-8db7-9e3085f84184-istio-token\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.503091 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.503073 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdjvb\" (UniqueName: \"kubernetes.io/projected/57f05af9-1c21-479d-8db7-9e3085f84184-kube-api-access-fdjvb\") pod \"e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.642422 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.642334 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:48.770462 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.770437 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk"] Jul 29 01:30:48.772408 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:30:48.772379 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f05af9_1c21_479d_8db7_9e3085f84184.slice/crio-72774f9c44545da3c0ec621dbce8ab8f1380d7d0dc360d71f58a0aa8ef29be5b WatchSource:0}: Error finding container 72774f9c44545da3c0ec621dbce8ab8f1380d7d0dc360d71f58a0aa8ef29be5b: Status 404 returned error can't find the container with id 72774f9c44545da3c0ec621dbce8ab8f1380d7d0dc360d71f58a0aa8ef29be5b Jul 29 01:30:48.774390 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.774360 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:30:48.774453 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.774424 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:30:48.774489 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:48.774453 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:30:49.088451 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:49.088415 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" event={"ID":"57f05af9-1c21-479d-8db7-9e3085f84184","Type":"ContainerStarted","Data":"335b2414384dbbd3fad09ed0b45534572144803b0b3551cc4fe8e8e19fd603fb"} Jul 29 01:30:49.088625 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:49.088470 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" event={"ID":"57f05af9-1c21-479d-8db7-9e3085f84184","Type":"ContainerStarted","Data":"72774f9c44545da3c0ec621dbce8ab8f1380d7d0dc360d71f58a0aa8ef29be5b"} Jul 29 01:30:49.110769 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:49.110708 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" podStartSLOduration=1.110690089 podStartE2EDuration="1.110690089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:30:49.108719421 +0000 UTC m=+2199.379363728" watchObservedRunningTime="2026-07-29 01:30:49.110690089 +0000 UTC m=+2199.381334397" Jul 29 01:30:49.643418 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:49.643391 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:49.648189 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:49.648167 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:50.094100 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:50.094071 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:30:50.094907 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:30:50.094887 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:31:04.165978 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:04.165944 2543 generic.go:356] "Generic (PLEG): container finished" podID="57f05af9-1c21-479d-8db7-9e3085f84184" containerID="335b2414384dbbd3fad09ed0b45534572144803b0b3551cc4fe8e8e19fd603fb" exitCode=0 Jul 29 01:31:04.166410 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:04.166018 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" event={"ID":"57f05af9-1c21-479d-8db7-9e3085f84184","Type":"ContainerDied","Data":"335b2414384dbbd3fad09ed0b45534572144803b0b3551cc4fe8e8e19fd603fb"} Jul 29 01:31:04.166410 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:04.166398 2543 scope.go:117] "RemoveContainer" containerID="335b2414384dbbd3fad09ed0b45534572144803b0b3551cc4fe8e8e19fd603fb" Jul 29 01:31:04.167367 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:04.167343 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:31:04.167425 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:04.167407 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:31:04.167460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:04.167437 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:31:05.094247 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:05.094206 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:31:05.172948 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:05.172912 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" event={"ID":"57f05af9-1c21-479d-8db7-9e3085f84184","Type":"ContainerStarted","Data":"fbae1e0db591eddcaddecb505863557b5d4132a3e736e012a16e8cee433ca7cd"} Jul 29 01:31:05.643152 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:05.643111 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:31:05.644395 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:05.644356 2543 patch_prober.go:28] interesting pod/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 29 01:31:05.644545 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:05.644426 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" podUID="57f05af9-1c21-479d-8db7-9e3085f84184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 29 01:31:06.642990 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:06.642954 2543 patch_prober.go:28] interesting pod/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 29 01:31:06.643494 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:06.643008 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" podUID="57f05af9-1c21-479d-8db7-9e3085f84184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 29 01:31:07.643300 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:07.643261 2543 patch_prober.go:28] interesting pod/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 29 01:31:07.643684 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:07.643324 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" podUID="57f05af9-1c21-479d-8db7-9e3085f84184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 29 01:31:08.643534 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:08.643498 2543 patch_prober.go:28] interesting pod/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 29 01:31:08.643945 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:08.643555 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" podUID="57f05af9-1c21-479d-8db7-9e3085f84184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 29 01:31:09.646683 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:09.646652 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:31:09.647549 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:09.647064 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:31:09.647646 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:09.647623 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:31:55.371726 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:55.371691 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f46b2363-lh8hm"] Jul 29 01:31:55.375621 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:55.375598 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f46b2363-lh8hm" Jul 29 01:31:55.377883 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:55.377794 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjvns\"" Jul 29 01:31:55.377883 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:55.377851 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 01:31:55.382673 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:55.382651 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f46b2363-lh8hm"] Jul 29 01:31:55.505053 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:55.505023 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5e87647-8abf-4f8d-b95a-8c33d36e22fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f46b2363-lh8hm\" (UID: \"a5e87647-8abf-4f8d-b95a-8c33d36e22fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f46b2363-lh8hm" Jul 29 01:31:55.606658 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:55.606622 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5e87647-8abf-4f8d-b95a-8c33d36e22fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f46b2363-lh8hm\" (UID: \"a5e87647-8abf-4f8d-b95a-8c33d36e22fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f46b2363-lh8hm" Jul 29 01:31:55.607191 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:55.607170 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5e87647-8abf-4f8d-b95a-8c33d36e22fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f46b2363-lh8hm\" (UID: \"a5e87647-8abf-4f8d-b95a-8c33d36e22fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f46b2363-lh8hm" Jul 29 01:31:55.688406 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:55.688333 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f46b2363-lh8hm" Jul 29 01:31:55.813278 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:55.813244 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f46b2363-lh8hm"] Jul 29 01:31:56.420771 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:56.420734 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f46b2363-lh8hm" event={"ID":"a5e87647-8abf-4f8d-b95a-8c33d36e22fa","Type":"ContainerStarted","Data":"cd123f1f3b33637e09f7b6c2aa6029de0852afb777331d98e748fcdb1cdd23f9"} Jul 29 01:31:56.421167 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:56.420787 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f46b2363-lh8hm" event={"ID":"a5e87647-8abf-4f8d-b95a-8c33d36e22fa","Type":"ContainerStarted","Data":"89b1478f397f357b484e2cada7756e8a6232019e0de14963309284246a0e8e56"} Jul 29 01:31:56.436411 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:56.436364 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f46b2363-lh8hm" podStartSLOduration=1.436349517 podStartE2EDuration="1.436349517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:31:56.434798114 +0000 UTC m=+2266.705442438" watchObservedRunningTime="2026-07-29 01:31:56.436349517 +0000 UTC m=+2266.706993824" Jul 29 01:31:57.427144 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:57.427111 2543 generic.go:356] "Generic (PLEG): container finished" podID="a5e87647-8abf-4f8d-b95a-8c33d36e22fa" containerID="cd123f1f3b33637e09f7b6c2aa6029de0852afb777331d98e748fcdb1cdd23f9" exitCode=0 Jul 29 01:31:57.427532 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:57.427177 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f46b2363-lh8hm" event={"ID":"a5e87647-8abf-4f8d-b95a-8c33d36e22fa","Type":"ContainerDied","Data":"cd123f1f3b33637e09f7b6c2aa6029de0852afb777331d98e748fcdb1cdd23f9"} Jul 29 01:31:58.566842 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:58.566805 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f46b2363-lh8hm" Jul 29 01:31:58.735341 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:58.735234 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5e87647-8abf-4f8d-b95a-8c33d36e22fa-maas-api-service-ca\") pod \"a5e87647-8abf-4f8d-b95a-8c33d36e22fa\" (UID: \"a5e87647-8abf-4f8d-b95a-8c33d36e22fa\") " Jul 29 01:31:58.735550 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:58.735529 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e87647-8abf-4f8d-b95a-8c33d36e22fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a5e87647-8abf-4f8d-b95a-8c33d36e22fa" (UID: "a5e87647-8abf-4f8d-b95a-8c33d36e22fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:31:58.836306 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:58.836261 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5e87647-8abf-4f8d-b95a-8c33d36e22fa-maas-api-service-ca\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:31:59.438763 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:59.438732 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f46b2363-lh8hm" Jul 29 01:31:59.438967 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:59.438732 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f46b2363-lh8hm" event={"ID":"a5e87647-8abf-4f8d-b95a-8c33d36e22fa","Type":"ContainerDied","Data":"89b1478f397f357b484e2cada7756e8a6232019e0de14963309284246a0e8e56"} Jul 29 01:31:59.438967 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:31:59.438859 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89b1478f397f357b484e2cada7756e8a6232019e0de14963309284246a0e8e56" Jul 29 01:32:10.347747 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:32:10.347711 2543 scope.go:117] "RemoveContainer" containerID="7b39cf3670101a0fbbf24adebb5f8a6aec772067cbe5afb82466cb738ebfac49" Jul 29 01:32:11.080453 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:32:11.080411 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f46b2363-lh8hm"] Jul 29 01:32:11.082665 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:32:11.082640 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f46b2363-lh8hm"] Jul 29 01:32:12.343289 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:32:12.343246 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5e87647-8abf-4f8d-b95a-8c33d36e22fa" path="/var/lib/kubelet/pods/a5e87647-8abf-4f8d-b95a-8c33d36e22fa/volumes" Jul 29 01:33:01.930782 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:01.930750 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk"] Jul 29 01:33:01.931313 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:01.931040 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" podUID="57f05af9-1c21-479d-8db7-9e3085f84184" containerName="istio-proxy" containerID="cri-o://fbae1e0db591eddcaddecb505863557b5d4132a3e736e012a16e8cee433ca7cd" gracePeriod=30 Jul 29 01:33:07.176020 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.175992 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:33:07.229033 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.228953 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-istio-envoy\") pod \"57f05af9-1c21-479d-8db7-9e3085f84184\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " Jul 29 01:33:07.229033 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.229015 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f05af9-1c21-479d-8db7-9e3085f84184-istio-podinfo\") pod \"57f05af9-1c21-479d-8db7-9e3085f84184\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " Jul 29 01:33:07.229257 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.229110 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-workload-certs\") pod \"57f05af9-1c21-479d-8db7-9e3085f84184\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " Jul 29 01:33:07.229257 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.229134 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-istio-data\") pod \"57f05af9-1c21-479d-8db7-9e3085f84184\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " Jul 29 01:33:07.229257 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.229165 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f05af9-1c21-479d-8db7-9e3085f84184-istio-token\") pod \"57f05af9-1c21-479d-8db7-9e3085f84184\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " Jul 29 01:33:07.229257 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.229188 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-credential-socket\") pod \"57f05af9-1c21-479d-8db7-9e3085f84184\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " Jul 29 01:33:07.229257 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.229214 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-workload-socket\") pod \"57f05af9-1c21-479d-8db7-9e3085f84184\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " Jul 29 01:33:07.229498 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.229267 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f05af9-1c21-479d-8db7-9e3085f84184-istiod-ca-cert\") pod \"57f05af9-1c21-479d-8db7-9e3085f84184\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " Jul 29 01:33:07.229498 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.229296 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdjvb\" (UniqueName: \"kubernetes.io/projected/57f05af9-1c21-479d-8db7-9e3085f84184-kube-api-access-fdjvb\") pod \"57f05af9-1c21-479d-8db7-9e3085f84184\" (UID: \"57f05af9-1c21-479d-8db7-9e3085f84184\") " Jul 29 01:33:07.229498 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.229432 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "57f05af9-1c21-479d-8db7-9e3085f84184" (UID: "57f05af9-1c21-479d-8db7-9e3085f84184"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:33:07.229651 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.229605 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-workload-certs\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:33:07.229712 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.229655 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "57f05af9-1c21-479d-8db7-9e3085f84184" (UID: "57f05af9-1c21-479d-8db7-9e3085f84184"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:33:07.229776 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.229715 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "57f05af9-1c21-479d-8db7-9e3085f84184" (UID: "57f05af9-1c21-479d-8db7-9e3085f84184"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:33:07.229965 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.229943 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f05af9-1c21-479d-8db7-9e3085f84184-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "57f05af9-1c21-479d-8db7-9e3085f84184" (UID: "57f05af9-1c21-479d-8db7-9e3085f84184"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:33:07.230055 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.230035 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-istio-data" (OuterVolumeSpecName: "istio-data") pod "57f05af9-1c21-479d-8db7-9e3085f84184" (UID: "57f05af9-1c21-479d-8db7-9e3085f84184"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:33:07.231586 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.231554 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f05af9-1c21-479d-8db7-9e3085f84184-istio-token" (OuterVolumeSpecName: "istio-token") pod "57f05af9-1c21-479d-8db7-9e3085f84184" (UID: "57f05af9-1c21-479d-8db7-9e3085f84184"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:33:07.231586 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.231574 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f05af9-1c21-479d-8db7-9e3085f84184-kube-api-access-fdjvb" (OuterVolumeSpecName: "kube-api-access-fdjvb") pod "57f05af9-1c21-479d-8db7-9e3085f84184" (UID: "57f05af9-1c21-479d-8db7-9e3085f84184"). InnerVolumeSpecName "kube-api-access-fdjvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:33:07.231767 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.231644 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "57f05af9-1c21-479d-8db7-9e3085f84184" (UID: "57f05af9-1c21-479d-8db7-9e3085f84184"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:33:07.231767 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.231646 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57f05af9-1c21-479d-8db7-9e3085f84184-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "57f05af9-1c21-479d-8db7-9e3085f84184" (UID: "57f05af9-1c21-479d-8db7-9e3085f84184"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 01:33:07.330924 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.330891 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-istio-data\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:33:07.330924 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.330919 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f05af9-1c21-479d-8db7-9e3085f84184-istio-token\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:33:07.330924 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.330928 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-credential-socket\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:33:07.331151 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.330936 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-workload-socket\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:33:07.331151 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.330946 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f05af9-1c21-479d-8db7-9e3085f84184-istiod-ca-cert\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:33:07.331151 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.330955 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdjvb\" (UniqueName: \"kubernetes.io/projected/57f05af9-1c21-479d-8db7-9e3085f84184-kube-api-access-fdjvb\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:33:07.331151 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.330964 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f05af9-1c21-479d-8db7-9e3085f84184-istio-envoy\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:33:07.331151 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.330973 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f05af9-1c21-479d-8db7-9e3085f84184-istio-podinfo\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:33:07.786050 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.786015 2543 generic.go:356] "Generic (PLEG): container finished" podID="57f05af9-1c21-479d-8db7-9e3085f84184" containerID="fbae1e0db591eddcaddecb505863557b5d4132a3e736e012a16e8cee433ca7cd" exitCode=0 Jul 29 01:33:07.786212 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.786079 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" Jul 29 01:33:07.786212 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.786093 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" event={"ID":"57f05af9-1c21-479d-8db7-9e3085f84184","Type":"ContainerDied","Data":"fbae1e0db591eddcaddecb505863557b5d4132a3e736e012a16e8cee433ca7cd"} Jul 29 01:33:07.786212 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.786182 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk" event={"ID":"57f05af9-1c21-479d-8db7-9e3085f84184","Type":"ContainerDied","Data":"72774f9c44545da3c0ec621dbce8ab8f1380d7d0dc360d71f58a0aa8ef29be5b"} Jul 29 01:33:07.786407 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.786213 2543 scope.go:117] "RemoveContainer" containerID="fbae1e0db591eddcaddecb505863557b5d4132a3e736e012a16e8cee433ca7cd" Jul 29 01:33:07.795484 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.795468 2543 scope.go:117] "RemoveContainer" containerID="335b2414384dbbd3fad09ed0b45534572144803b0b3551cc4fe8e8e19fd603fb" Jul 29 01:33:07.803463 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.803447 2543 scope.go:117] "RemoveContainer" containerID="fbae1e0db591eddcaddecb505863557b5d4132a3e736e012a16e8cee433ca7cd" Jul 29 01:33:07.803699 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:33:07.803682 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbae1e0db591eddcaddecb505863557b5d4132a3e736e012a16e8cee433ca7cd\": container with ID starting with fbae1e0db591eddcaddecb505863557b5d4132a3e736e012a16e8cee433ca7cd not found: ID does not exist" containerID="fbae1e0db591eddcaddecb505863557b5d4132a3e736e012a16e8cee433ca7cd" Jul 29 01:33:07.803751 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.803706 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbae1e0db591eddcaddecb505863557b5d4132a3e736e012a16e8cee433ca7cd"} err="failed to get container status \"fbae1e0db591eddcaddecb505863557b5d4132a3e736e012a16e8cee433ca7cd\": rpc error: code = NotFound desc = could not find container \"fbae1e0db591eddcaddecb505863557b5d4132a3e736e012a16e8cee433ca7cd\": container with ID starting with fbae1e0db591eddcaddecb505863557b5d4132a3e736e012a16e8cee433ca7cd not found: ID does not exist" Jul 29 01:33:07.803751 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.803722 2543 scope.go:117] "RemoveContainer" containerID="335b2414384dbbd3fad09ed0b45534572144803b0b3551cc4fe8e8e19fd603fb" Jul 29 01:33:07.803962 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:33:07.803942 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"335b2414384dbbd3fad09ed0b45534572144803b0b3551cc4fe8e8e19fd603fb\": container with ID starting with 335b2414384dbbd3fad09ed0b45534572144803b0b3551cc4fe8e8e19fd603fb not found: ID does not exist" containerID="335b2414384dbbd3fad09ed0b45534572144803b0b3551cc4fe8e8e19fd603fb" Jul 29 01:33:07.804017 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.803972 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"335b2414384dbbd3fad09ed0b45534572144803b0b3551cc4fe8e8e19fd603fb"} err="failed to get container status \"335b2414384dbbd3fad09ed0b45534572144803b0b3551cc4fe8e8e19fd603fb\": rpc error: code = NotFound desc = could not find container \"335b2414384dbbd3fad09ed0b45534572144803b0b3551cc4fe8e8e19fd603fb\": container with ID starting with 335b2414384dbbd3fad09ed0b45534572144803b0b3551cc4fe8e8e19fd603fb not found: ID does not exist" Jul 29 01:33:07.807514 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.807491 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk"] Jul 29 01:33:07.814215 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:07.814187 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f46b2363-openshift-default-bcb9969b4-n6jxk"] Jul 29 01:33:08.343561 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:08.343533 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57f05af9-1c21-479d-8db7-9e3085f84184" path="/var/lib/kubelet/pods/57f05af9-1c21-479d-8db7-9e3085f84184/volumes" Jul 29 01:33:10.366607 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:10.366575 2543 scope.go:117] "RemoveContainer" containerID="01e6ae11b3143293734693f1c6ce2aab073d74d8a49045bec9b8c755d483a00e" Jul 29 01:33:10.374640 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:10.374621 2543 scope.go:117] "RemoveContainer" containerID="7b2e12f2a644b574699b9eb5f9081f25bff60515ca743f546022e6b29d4474ac" Jul 29 01:33:56.596677 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.596567 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7"] Jul 29 01:33:56.597175 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.597051 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57f05af9-1c21-479d-8db7-9e3085f84184" containerName="istio-proxy" Jul 29 01:33:56.597175 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.597068 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="57f05af9-1c21-479d-8db7-9e3085f84184" containerName="istio-proxy" Jul 29 01:33:56.597175 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.597085 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5e87647-8abf-4f8d-b95a-8c33d36e22fa" containerName="revoke-keys" Jul 29 01:33:56.597175 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.597094 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5e87647-8abf-4f8d-b95a-8c33d36e22fa" containerName="revoke-keys" Jul 29 01:33:56.597334 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.597208 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="57f05af9-1c21-479d-8db7-9e3085f84184" containerName="istio-proxy" Jul 29 01:33:56.597334 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.597219 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5e87647-8abf-4f8d-b95a-8c33d36e22fa" containerName="revoke-keys" Jul 29 01:33:56.600072 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.600057 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.602717 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.602697 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6afea2-openshift-default-dockercfg-mmddh\"" Jul 29 01:33:56.612927 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.612902 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7"] Jul 29 01:33:56.650510 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.650481 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-podinfo\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.650657 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.650525 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-envoy\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.650657 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.650580 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-data\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.650657 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.650615 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-workload-certs\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.650657 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.650656 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istiod-ca-cert\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.650805 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.650675 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-workload-socket\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.650805 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.650696 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll76v\" (UniqueName: \"kubernetes.io/projected/0039f4e3-41e9-4963-9ba8-15e597d7d40b-kube-api-access-ll76v\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.650805 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.650728 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-token\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.650805 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.650789 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-credential-socket\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.751288 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.751250 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-podinfo\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.751467 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.751305 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-envoy\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.751467 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.751331 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-data\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.751467 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.751356 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-workload-certs\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.751467 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.751402 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istiod-ca-cert\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.751467 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.751427 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-workload-socket\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.751467 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.751449 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll76v\" (UniqueName: \"kubernetes.io/projected/0039f4e3-41e9-4963-9ba8-15e597d7d40b-kube-api-access-ll76v\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.751791 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.751498 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-token\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.751791 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.751552 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-credential-socket\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.751920 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.751811 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-data\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.752147 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.752120 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-credential-socket\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.752343 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.752172 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-workload-certs\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.752343 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.752330 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-workload-socket\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.752831 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.752792 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istiod-ca-cert\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.754094 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.754072 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-envoy\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.754228 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.754198 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-podinfo\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.759649 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.759623 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-token\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.759757 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.759735 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll76v\" (UniqueName: \"kubernetes.io/projected/0039f4e3-41e9-4963-9ba8-15e597d7d40b-kube-api-access-ll76v\") pod \"e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:56.910564 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:56.910463 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:57.048172 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:57.048144 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7"] Jul 29 01:33:57.049808 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:33:57.049780 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0039f4e3_41e9_4963_9ba8_15e597d7d40b.slice/crio-d5223f1fd31bc4ac33ac8069d9097e363740e85462cfee8d8f1a9893599b91e6 WatchSource:0}: Error finding container d5223f1fd31bc4ac33ac8069d9097e363740e85462cfee8d8f1a9893599b91e6: Status 404 returned error can't find the container with id d5223f1fd31bc4ac33ac8069d9097e363740e85462cfee8d8f1a9893599b91e6 Jul 29 01:33:57.052034 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:57.052001 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:33:57.052124 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:57.052066 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:33:57.052124 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:57.052092 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:33:58.026426 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:58.026386 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" event={"ID":"0039f4e3-41e9-4963-9ba8-15e597d7d40b","Type":"ContainerStarted","Data":"3e63da219f956d9e0c9a776f02691edbd8ddd8f71137b9074c7eefd3cdf05cb2"} Jul 29 01:33:58.026915 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:58.026455 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" event={"ID":"0039f4e3-41e9-4963-9ba8-15e597d7d40b","Type":"ContainerStarted","Data":"d5223f1fd31bc4ac33ac8069d9097e363740e85462cfee8d8f1a9893599b91e6"} Jul 29 01:33:58.046529 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:58.046483 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" podStartSLOduration=2.046470072 podStartE2EDuration="2.046470072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:33:58.045260001 +0000 UTC m=+2388.315904325" watchObservedRunningTime="2026-07-29 01:33:58.046470072 +0000 UTC m=+2388.317114378" Jul 29 01:33:58.911589 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:58.911551 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:58.916328 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:58.916301 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:59.030954 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:59.030923 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:33:59.031853 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:33:59.031835 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:34:10.401246 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:10.401203 2543 scope.go:117] "RemoveContainer" containerID="f1fa2efc2ab59af25b96aeae21433b8012fbae6aa0b224f072244ecefebc0d15" Jul 29 01:34:10.690356 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:10.690264 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 01:34:11.025721 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:11.025638 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 01:34:14.032608 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:14.032572 2543 patch_prober.go:28] interesting pod/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 29 01:34:14.033042 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:14.032640 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" podUID="0039f4e3-41e9-4963-9ba8-15e597d7d40b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 29 01:34:14.108229 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:14.108197 2543 generic.go:356] "Generic (PLEG): container finished" podID="0039f4e3-41e9-4963-9ba8-15e597d7d40b" containerID="3e63da219f956d9e0c9a776f02691edbd8ddd8f71137b9074c7eefd3cdf05cb2" exitCode=0 Jul 29 01:34:14.108386 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:14.108273 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" event={"ID":"0039f4e3-41e9-4963-9ba8-15e597d7d40b","Type":"ContainerDied","Data":"3e63da219f956d9e0c9a776f02691edbd8ddd8f71137b9074c7eefd3cdf05cb2"} Jul 29 01:34:14.108681 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:14.108658 2543 scope.go:117] "RemoveContainer" containerID="3e63da219f956d9e0c9a776f02691edbd8ddd8f71137b9074c7eefd3cdf05cb2" Jul 29 01:34:14.109735 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:14.109712 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:34:14.109790 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:14.109774 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:34:14.109845 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:14.109805 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:34:15.117796 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:15.117762 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" event={"ID":"0039f4e3-41e9-4963-9ba8-15e597d7d40b","Type":"ContainerStarted","Data":"f52f539c8f8ddf8fa99807ecaa2525b1419b7dd252b7622b0f630020a8276c03"} Jul 29 01:34:15.911155 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:15.911108 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:34:15.911378 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:15.911253 2543 patch_prober.go:28] interesting pod/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 29 01:34:15.911378 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:15.911322 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" podUID="0039f4e3-41e9-4963-9ba8-15e597d7d40b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 29 01:34:16.911123 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:16.911076 2543 patch_prober.go:28] interesting pod/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 29 01:34:16.911673 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:16.911147 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" podUID="0039f4e3-41e9-4963-9ba8-15e597d7d40b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 29 01:34:17.913338 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:17.913227 2543 patch_prober.go:28] interesting pod/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 29 01:34:17.913965 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:17.913361 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" podUID="0039f4e3-41e9-4963-9ba8-15e597d7d40b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 29 01:34:18.910939 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:18.910898 2543 patch_prober.go:28] interesting pod/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 29 01:34:18.911267 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:18.911237 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" podUID="0039f4e3-41e9-4963-9ba8-15e597d7d40b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 29 01:34:19.914338 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:19.914308 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:34:19.914777 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:19.914644 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:34:19.915259 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:19.915242 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:34:25.536556 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.536519 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb"] Jul 29 01:34:25.536964 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.536873 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57f05af9-1c21-479d-8db7-9e3085f84184" containerName="istio-proxy" Jul 29 01:34:25.536964 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.536885 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="57f05af9-1c21-479d-8db7-9e3085f84184" containerName="istio-proxy" Jul 29 01:34:25.536964 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.536950 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="57f05af9-1c21-479d-8db7-9e3085f84184" containerName="istio-proxy" Jul 29 01:34:25.540018 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.540002 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.542857 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.542838 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f5ec11-openshift-default-dockercfg-vrxg2\"" Jul 29 01:34:25.553110 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.553089 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb"] Jul 29 01:34:25.604768 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.604731 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c1a3571-1f3e-46f1-8361-0c84686fc618-istiod-ca-cert\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.604948 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.604847 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-data\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.604948 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.604921 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-workload-certs\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.605049 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.604967 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-envoy\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.605049 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.604984 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk29k\" (UniqueName: \"kubernetes.io/projected/6c1a3571-1f3e-46f1-8361-0c84686fc618-kube-api-access-bk29k\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.605134 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.605072 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-podinfo\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.605134 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.605106 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-workload-socket\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.605134 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.605128 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-credential-socket\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.605264 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.605143 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-token\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.706394 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.706353 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-data\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.706580 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.706407 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-workload-certs\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.706580 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.706440 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-envoy\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.706580 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.706464 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk29k\" (UniqueName: \"kubernetes.io/projected/6c1a3571-1f3e-46f1-8361-0c84686fc618-kube-api-access-bk29k\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.706580 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.706519 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-podinfo\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.706580 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.706551 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-workload-socket\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.706947 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.706572 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-credential-socket\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.707015 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.706973 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-token\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.707076 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.707012 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c1a3571-1f3e-46f1-8361-0c84686fc618-istiod-ca-cert\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.707076 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.706859 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-workload-certs\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.707076 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.706899 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-data\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.707247 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.707091 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-workload-socket\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.707247 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.707133 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-credential-socket\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.707729 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.707708 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c1a3571-1f3e-46f1-8361-0c84686fc618-istiod-ca-cert\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.709880 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.709670 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-envoy\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.709880 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.709847 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-podinfo\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.714802 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.714768 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-token\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.715001 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.714979 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk29k\" (UniqueName: \"kubernetes.io/projected/6c1a3571-1f3e-46f1-8361-0c84686fc618-kube-api-access-bk29k\") pod \"e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.850798 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.850772 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:25.988583 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.988558 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb"] Jul 29 01:34:25.990983 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:34:25.990955 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1a3571_1f3e_46f1_8361_0c84686fc618.slice/crio-ec516735d573fba1f466f722589d2ca7e8f3a0bddc4560f60ae65d58d4c08d6c WatchSource:0}: Error finding container ec516735d573fba1f466f722589d2ca7e8f3a0bddc4560f60ae65d58d4c08d6c: Status 404 returned error can't find the container with id ec516735d573fba1f466f722589d2ca7e8f3a0bddc4560f60ae65d58d4c08d6c Jul 29 01:34:25.993367 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.993336 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:34:25.993454 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.993401 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:34:25.993454 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:25.993431 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:34:26.171204 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:26.171114 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" event={"ID":"6c1a3571-1f3e-46f1-8361-0c84686fc618","Type":"ContainerStarted","Data":"415b64f05add9056b45610c067363a0a7ca4c48d029ce5f92da908b5a0c9a823"} Jul 29 01:34:26.171204 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:26.171162 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" event={"ID":"6c1a3571-1f3e-46f1-8361-0c84686fc618","Type":"ContainerStarted","Data":"ec516735d573fba1f466f722589d2ca7e8f3a0bddc4560f60ae65d58d4c08d6c"} Jul 29 01:34:26.194866 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:26.194807 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" podStartSLOduration=1.194791746 podStartE2EDuration="1.194791746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:34:26.193330011 +0000 UTC m=+2416.463974330" watchObservedRunningTime="2026-07-29 01:34:26.194791746 +0000 UTC m=+2416.465436052" Jul 29 01:34:26.850947 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:26.850907 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:27.856874 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:27.856835 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:28.181656 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:28.181580 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:28.182444 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:28.182425 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:43.580986 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:43.580038 2543 patch_prober.go:28] interesting pod/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": read tcp 10.132.0.2:56552->10.132.0.49:15021: read: connection reset by peer" start-of-body= Jul 29 01:34:43.580986 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:43.580119 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" podUID="6c1a3571-1f3e-46f1-8361-0c84686fc618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": read tcp 10.132.0.2:56552->10.132.0.49:15021: read: connection reset by peer" Jul 29 01:34:44.261142 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:44.261109 2543 generic.go:356] "Generic (PLEG): container finished" podID="6c1a3571-1f3e-46f1-8361-0c84686fc618" containerID="415b64f05add9056b45610c067363a0a7ca4c48d029ce5f92da908b5a0c9a823" exitCode=0 Jul 29 01:34:44.261308 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:44.261150 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" event={"ID":"6c1a3571-1f3e-46f1-8361-0c84686fc618","Type":"ContainerDied","Data":"415b64f05add9056b45610c067363a0a7ca4c48d029ce5f92da908b5a0c9a823"} Jul 29 01:34:44.261547 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:44.261534 2543 scope.go:117] "RemoveContainer" containerID="415b64f05add9056b45610c067363a0a7ca4c48d029ce5f92da908b5a0c9a823" Jul 29 01:34:44.262517 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:44.262491 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:34:44.262603 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:44.262558 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:34:44.262647 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:44.262597 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:34:45.268177 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:45.268139 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" event={"ID":"6c1a3571-1f3e-46f1-8361-0c84686fc618","Type":"ContainerStarted","Data":"3e1ab5dcaa4c9815da3f44c7e17aface4c7ef141f711710d1b66fa2915b5680c"} Jul 29 01:34:45.851987 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:45.851942 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:45.852197 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:45.852169 2543 patch_prober.go:28] interesting pod/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 29 01:34:45.852278 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:45.852228 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" podUID="6c1a3571-1f3e-46f1-8361-0c84686fc618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 29 01:34:46.851341 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:46.851307 2543 patch_prober.go:28] interesting pod/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 29 01:34:46.851725 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:46.851362 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" podUID="6c1a3571-1f3e-46f1-8361-0c84686fc618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 29 01:34:47.851734 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:47.851698 2543 patch_prober.go:28] interesting pod/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 29 01:34:47.852194 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:47.851757 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" podUID="6c1a3571-1f3e-46f1-8361-0c84686fc618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 29 01:34:48.851851 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:48.851791 2543 patch_prober.go:28] interesting pod/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 29 01:34:48.852566 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:48.851887 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" podUID="6c1a3571-1f3e-46f1-8361-0c84686fc618" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 29 01:34:49.854975 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:49.854943 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:49.855431 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:49.855228 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:34:49.855954 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:34:49.855937 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:36:08.709475 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:08.709396 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6afea2-dc7zc"] Jul 29 01:36:08.712723 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:08.712706 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6afea2-dc7zc" Jul 29 01:36:08.715141 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:08.715120 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 01:36:08.715141 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:08.715130 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjvns\"" Jul 29 01:36:08.720186 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:08.720123 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6afea2-dc7zc"] Jul 29 01:36:08.846903 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:08.846871 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9c36e4-a301-44c4-9ebe-1a7bf8639d34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6afea2-dc7zc\" (UID: \"ef9c36e4-a301-44c4-9ebe-1a7bf8639d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6afea2-dc7zc" Jul 29 01:36:08.947989 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:08.947951 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9c36e4-a301-44c4-9ebe-1a7bf8639d34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6afea2-dc7zc\" (UID: \"ef9c36e4-a301-44c4-9ebe-1a7bf8639d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6afea2-dc7zc" Jul 29 01:36:08.948562 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:08.948542 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9c36e4-a301-44c4-9ebe-1a7bf8639d34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6afea2-dc7zc\" (UID: \"ef9c36e4-a301-44c4-9ebe-1a7bf8639d34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6afea2-dc7zc" Jul 29 01:36:09.024203 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:09.024114 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6afea2-dc7zc" Jul 29 01:36:09.156360 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:09.156328 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6afea2-dc7zc"] Jul 29 01:36:09.158396 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:36:09.158367 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9c36e4_a301_44c4_9ebe_1a7bf8639d34.slice/crio-67611c99776e30a2f09f7b0cc1c7d7a9fcce24ec78a6501c655b16a93ddcbfb5 WatchSource:0}: Error finding container 67611c99776e30a2f09f7b0cc1c7d7a9fcce24ec78a6501c655b16a93ddcbfb5: Status 404 returned error can't find the container with id 67611c99776e30a2f09f7b0cc1c7d7a9fcce24ec78a6501c655b16a93ddcbfb5 Jul 29 01:36:09.685648 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:09.685564 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6afea2-dc7zc" event={"ID":"ef9c36e4-a301-44c4-9ebe-1a7bf8639d34","Type":"ContainerStarted","Data":"ca8db452f198072e02b430f160e09dd6b4044ea1d2256ebb77fb24f65772eade"} Jul 29 01:36:09.685648 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:09.685624 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6afea2-dc7zc" event={"ID":"ef9c36e4-a301-44c4-9ebe-1a7bf8639d34","Type":"ContainerStarted","Data":"67611c99776e30a2f09f7b0cc1c7d7a9fcce24ec78a6501c655b16a93ddcbfb5"} Jul 29 01:36:09.701239 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:09.701189 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6afea2-dc7zc" podStartSLOduration=1.701174468 podStartE2EDuration="1.701174468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:36:09.700047193 +0000 UTC m=+2519.970691500" watchObservedRunningTime="2026-07-29 01:36:09.701174468 +0000 UTC m=+2519.971818774" Jul 29 01:36:10.434492 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:10.434460 2543 scope.go:117] "RemoveContainer" containerID="d650c51bb35110251fcd7d5bde959e3d475c69e6698ef463e6a06555f1f08fcb" Jul 29 01:36:10.692054 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:10.691967 2543 generic.go:356] "Generic (PLEG): container finished" podID="ef9c36e4-a301-44c4-9ebe-1a7bf8639d34" containerID="ca8db452f198072e02b430f160e09dd6b4044ea1d2256ebb77fb24f65772eade" exitCode=0 Jul 29 01:36:10.692200 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:10.692059 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6afea2-dc7zc" event={"ID":"ef9c36e4-a301-44c4-9ebe-1a7bf8639d34","Type":"ContainerDied","Data":"ca8db452f198072e02b430f160e09dd6b4044ea1d2256ebb77fb24f65772eade"} Jul 29 01:36:11.830703 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:11.830675 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6afea2-dc7zc" Jul 29 01:36:11.978108 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:11.978021 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9c36e4-a301-44c4-9ebe-1a7bf8639d34-maas-api-service-ca\") pod \"ef9c36e4-a301-44c4-9ebe-1a7bf8639d34\" (UID: \"ef9c36e4-a301-44c4-9ebe-1a7bf8639d34\") " Jul 29 01:36:11.978387 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:11.978362 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9c36e4-a301-44c4-9ebe-1a7bf8639d34-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ef9c36e4-a301-44c4-9ebe-1a7bf8639d34" (UID: "ef9c36e4-a301-44c4-9ebe-1a7bf8639d34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:36:12.079567 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:12.079525 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9c36e4-a301-44c4-9ebe-1a7bf8639d34-maas-api-service-ca\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:36:12.703175 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:12.703081 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6afea2-dc7zc" Jul 29 01:36:12.703175 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:12.703103 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6afea2-dc7zc" event={"ID":"ef9c36e4-a301-44c4-9ebe-1a7bf8639d34","Type":"ContainerDied","Data":"67611c99776e30a2f09f7b0cc1c7d7a9fcce24ec78a6501c655b16a93ddcbfb5"} Jul 29 01:36:12.703175 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:12.703148 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67611c99776e30a2f09f7b0cc1c7d7a9fcce24ec78a6501c655b16a93ddcbfb5" Jul 29 01:36:19.518356 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:19.518306 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6afea2-dc7zc"] Jul 29 01:36:19.521517 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:19.521488 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6afea2-dc7zc"] Jul 29 01:36:20.344114 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:20.344078 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef9c36e4-a301-44c4-9ebe-1a7bf8639d34" path="/var/lib/kubelet/pods/ef9c36e4-a301-44c4-9ebe-1a7bf8639d34/volumes" Jul 29 01:36:25.908872 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:25.908838 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7"] Jul 29 01:36:25.909351 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:25.909072 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" podUID="0039f4e3-41e9-4963-9ba8-15e597d7d40b" containerName="istio-proxy" containerID="cri-o://f52f539c8f8ddf8fa99807ecaa2525b1419b7dd252b7622b0f630020a8276c03" gracePeriod=30 Jul 29 01:36:31.165376 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.165352 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:36:31.242435 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.242343 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-workload-socket\") pod \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " Jul 29 01:36:31.242435 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.242383 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-podinfo\") pod \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " Jul 29 01:36:31.242435 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.242405 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-workload-certs\") pod \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " Jul 29 01:36:31.242435 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.242420 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-token\") pod \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " Jul 29 01:36:31.242435 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.242438 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-envoy\") pod \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " Jul 29 01:36:31.242851 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.242479 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istiod-ca-cert\") pod \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " Jul 29 01:36:31.242851 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.242561 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ll76v\" (UniqueName: \"kubernetes.io/projected/0039f4e3-41e9-4963-9ba8-15e597d7d40b-kube-api-access-ll76v\") pod \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " Jul 29 01:36:31.242851 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.242589 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-data\") pod \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " Jul 29 01:36:31.242851 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.242612 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-credential-socket\") pod \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\" (UID: \"0039f4e3-41e9-4963-9ba8-15e597d7d40b\") " Jul 29 01:36:31.242851 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.242678 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0039f4e3-41e9-4963-9ba8-15e597d7d40b" (UID: "0039f4e3-41e9-4963-9ba8-15e597d7d40b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:36:31.242851 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.242710 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0039f4e3-41e9-4963-9ba8-15e597d7d40b" (UID: "0039f4e3-41e9-4963-9ba8-15e597d7d40b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:36:31.243148 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.242959 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0039f4e3-41e9-4963-9ba8-15e597d7d40b" (UID: "0039f4e3-41e9-4963-9ba8-15e597d7d40b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:36:31.243148 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.242987 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-workload-socket\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:36:31.243148 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.243008 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-workload-certs\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:36:31.243148 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.243048 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0039f4e3-41e9-4963-9ba8-15e597d7d40b" (UID: "0039f4e3-41e9-4963-9ba8-15e597d7d40b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:36:31.243401 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.243369 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-data" (OuterVolumeSpecName: "istio-data") pod "0039f4e3-41e9-4963-9ba8-15e597d7d40b" (UID: "0039f4e3-41e9-4963-9ba8-15e597d7d40b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:36:31.244713 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.244679 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0039f4e3-41e9-4963-9ba8-15e597d7d40b-kube-api-access-ll76v" (OuterVolumeSpecName: "kube-api-access-ll76v") pod "0039f4e3-41e9-4963-9ba8-15e597d7d40b" (UID: "0039f4e3-41e9-4963-9ba8-15e597d7d40b"). InnerVolumeSpecName "kube-api-access-ll76v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:36:31.244856 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.244725 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0039f4e3-41e9-4963-9ba8-15e597d7d40b" (UID: "0039f4e3-41e9-4963-9ba8-15e597d7d40b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 01:36:31.245061 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.245038 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0039f4e3-41e9-4963-9ba8-15e597d7d40b" (UID: "0039f4e3-41e9-4963-9ba8-15e597d7d40b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:36:31.245061 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.245041 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-token" (OuterVolumeSpecName: "istio-token") pod "0039f4e3-41e9-4963-9ba8-15e597d7d40b" (UID: "0039f4e3-41e9-4963-9ba8-15e597d7d40b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:36:31.343980 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.343943 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-podinfo\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:36:31.343980 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.343979 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-token\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:36:31.344212 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.343994 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-envoy\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:36:31.344212 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.344005 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istiod-ca-cert\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:36:31.344212 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.344016 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll76v\" (UniqueName: \"kubernetes.io/projected/0039f4e3-41e9-4963-9ba8-15e597d7d40b-kube-api-access-ll76v\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:36:31.344212 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.344028 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-istio-data\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:36:31.344212 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.344039 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0039f4e3-41e9-4963-9ba8-15e597d7d40b-credential-socket\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:36:31.795698 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.795668 2543 generic.go:356] "Generic (PLEG): container finished" podID="0039f4e3-41e9-4963-9ba8-15e597d7d40b" containerID="f52f539c8f8ddf8fa99807ecaa2525b1419b7dd252b7622b0f630020a8276c03" exitCode=0 Jul 29 01:36:31.795905 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.795714 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" Jul 29 01:36:31.795905 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.795752 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" event={"ID":"0039f4e3-41e9-4963-9ba8-15e597d7d40b","Type":"ContainerDied","Data":"f52f539c8f8ddf8fa99807ecaa2525b1419b7dd252b7622b0f630020a8276c03"} Jul 29 01:36:31.795905 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.795808 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7" event={"ID":"0039f4e3-41e9-4963-9ba8-15e597d7d40b","Type":"ContainerDied","Data":"d5223f1fd31bc4ac33ac8069d9097e363740e85462cfee8d8f1a9893599b91e6"} Jul 29 01:36:31.795905 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.795847 2543 scope.go:117] "RemoveContainer" containerID="f52f539c8f8ddf8fa99807ecaa2525b1419b7dd252b7622b0f630020a8276c03" Jul 29 01:36:31.805371 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.805352 2543 scope.go:117] "RemoveContainer" containerID="3e63da219f956d9e0c9a776f02691edbd8ddd8f71137b9074c7eefd3cdf05cb2" Jul 29 01:36:31.812943 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.812923 2543 scope.go:117] "RemoveContainer" containerID="f52f539c8f8ddf8fa99807ecaa2525b1419b7dd252b7622b0f630020a8276c03" Jul 29 01:36:31.813175 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:36:31.813156 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52f539c8f8ddf8fa99807ecaa2525b1419b7dd252b7622b0f630020a8276c03\": container with ID starting with f52f539c8f8ddf8fa99807ecaa2525b1419b7dd252b7622b0f630020a8276c03 not found: ID does not exist" containerID="f52f539c8f8ddf8fa99807ecaa2525b1419b7dd252b7622b0f630020a8276c03" Jul 29 01:36:31.813222 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.813183 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52f539c8f8ddf8fa99807ecaa2525b1419b7dd252b7622b0f630020a8276c03"} err="failed to get container status \"f52f539c8f8ddf8fa99807ecaa2525b1419b7dd252b7622b0f630020a8276c03\": rpc error: code = NotFound desc = could not find container \"f52f539c8f8ddf8fa99807ecaa2525b1419b7dd252b7622b0f630020a8276c03\": container with ID starting with f52f539c8f8ddf8fa99807ecaa2525b1419b7dd252b7622b0f630020a8276c03 not found: ID does not exist" Jul 29 01:36:31.813222 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.813201 2543 scope.go:117] "RemoveContainer" containerID="3e63da219f956d9e0c9a776f02691edbd8ddd8f71137b9074c7eefd3cdf05cb2" Jul 29 01:36:31.813451 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:36:31.813431 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e63da219f956d9e0c9a776f02691edbd8ddd8f71137b9074c7eefd3cdf05cb2\": container with ID starting with 3e63da219f956d9e0c9a776f02691edbd8ddd8f71137b9074c7eefd3cdf05cb2 not found: ID does not exist" containerID="3e63da219f956d9e0c9a776f02691edbd8ddd8f71137b9074c7eefd3cdf05cb2" Jul 29 01:36:31.813491 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.813456 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e63da219f956d9e0c9a776f02691edbd8ddd8f71137b9074c7eefd3cdf05cb2"} err="failed to get container status \"3e63da219f956d9e0c9a776f02691edbd8ddd8f71137b9074c7eefd3cdf05cb2\": rpc error: code = NotFound desc = could not find container \"3e63da219f956d9e0c9a776f02691edbd8ddd8f71137b9074c7eefd3cdf05cb2\": container with ID starting with 3e63da219f956d9e0c9a776f02691edbd8ddd8f71137b9074c7eefd3cdf05cb2 not found: ID does not exist" Jul 29 01:36:31.816846 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.816810 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7"] Jul 29 01:36:31.820419 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:31.820399 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6afea2-openshift-default-5c4dccdc76-5q8v7"] Jul 29 01:36:32.343169 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:32.343135 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0039f4e3-41e9-4963-9ba8-15e597d7d40b" path="/var/lib/kubelet/pods/0039f4e3-41e9-4963-9ba8-15e597d7d40b/volumes" Jul 29 01:36:43.916548 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:43.916512 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb"] Jul 29 01:36:43.917004 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:43.916775 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" podUID="6c1a3571-1f3e-46f1-8361-0c84686fc618" containerName="istio-proxy" containerID="cri-o://3e1ab5dcaa4c9815da3f44c7e17aface4c7ef141f711710d1b66fa2915b5680c" gracePeriod=30 Jul 29 01:36:49.172722 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.172696 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:36:49.297153 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.297079 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-data\") pod \"6c1a3571-1f3e-46f1-8361-0c84686fc618\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " Jul 29 01:36:49.297153 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.297109 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-envoy\") pod \"6c1a3571-1f3e-46f1-8361-0c84686fc618\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " Jul 29 01:36:49.297153 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.297125 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-workload-socket\") pod \"6c1a3571-1f3e-46f1-8361-0c84686fc618\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " Jul 29 01:36:49.297410 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.297157 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-podinfo\") pod \"6c1a3571-1f3e-46f1-8361-0c84686fc618\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " Jul 29 01:36:49.297410 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.297188 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bk29k\" (UniqueName: \"kubernetes.io/projected/6c1a3571-1f3e-46f1-8361-0c84686fc618-kube-api-access-bk29k\") pod \"6c1a3571-1f3e-46f1-8361-0c84686fc618\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " Jul 29 01:36:49.297410 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.297206 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c1a3571-1f3e-46f1-8361-0c84686fc618-istiod-ca-cert\") pod \"6c1a3571-1f3e-46f1-8361-0c84686fc618\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " Jul 29 01:36:49.297410 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.297250 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-credential-socket\") pod \"6c1a3571-1f3e-46f1-8361-0c84686fc618\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " Jul 29 01:36:49.297410 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.297268 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-token\") pod \"6c1a3571-1f3e-46f1-8361-0c84686fc618\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " Jul 29 01:36:49.297675 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.297424 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6c1a3571-1f3e-46f1-8361-0c84686fc618" (UID: "6c1a3571-1f3e-46f1-8361-0c84686fc618"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:36:49.297730 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.297682 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6c1a3571-1f3e-46f1-8361-0c84686fc618" (UID: "6c1a3571-1f3e-46f1-8361-0c84686fc618"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:36:49.297730 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.297694 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c1a3571-1f3e-46f1-8361-0c84686fc618-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6c1a3571-1f3e-46f1-8361-0c84686fc618" (UID: "6c1a3571-1f3e-46f1-8361-0c84686fc618"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:36:49.297872 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.297754 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-workload-certs\") pod \"6c1a3571-1f3e-46f1-8361-0c84686fc618\" (UID: \"6c1a3571-1f3e-46f1-8361-0c84686fc618\") " Jul 29 01:36:49.298084 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.298048 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6c1a3571-1f3e-46f1-8361-0c84686fc618" (UID: "6c1a3571-1f3e-46f1-8361-0c84686fc618"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:36:49.298084 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.298064 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-data" (OuterVolumeSpecName: "istio-data") pod "6c1a3571-1f3e-46f1-8361-0c84686fc618" (UID: "6c1a3571-1f3e-46f1-8361-0c84686fc618"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:36:49.298212 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.298199 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-credential-socket\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:36:49.298252 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.298220 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-workload-certs\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:36:49.298252 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.298235 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-data\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:36:49.298252 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.298249 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-workload-socket\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:36:49.298341 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.298263 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c1a3571-1f3e-46f1-8361-0c84686fc618-istiod-ca-cert\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:36:49.299582 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.299559 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-token" (OuterVolumeSpecName: "istio-token") pod "6c1a3571-1f3e-46f1-8361-0c84686fc618" (UID: "6c1a3571-1f3e-46f1-8361-0c84686fc618"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:36:49.299677 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.299587 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1a3571-1f3e-46f1-8361-0c84686fc618-kube-api-access-bk29k" (OuterVolumeSpecName: "kube-api-access-bk29k") pod "6c1a3571-1f3e-46f1-8361-0c84686fc618" (UID: "6c1a3571-1f3e-46f1-8361-0c84686fc618"). InnerVolumeSpecName "kube-api-access-bk29k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:36:49.299776 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.299759 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6c1a3571-1f3e-46f1-8361-0c84686fc618" (UID: "6c1a3571-1f3e-46f1-8361-0c84686fc618"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:36:49.299861 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.299789 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6c1a3571-1f3e-46f1-8361-0c84686fc618" (UID: "6c1a3571-1f3e-46f1-8361-0c84686fc618"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 01:36:49.399037 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.399002 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-envoy\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:36:49.399037 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.399034 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-podinfo\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:36:49.399248 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.399047 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk29k\" (UniqueName: \"kubernetes.io/projected/6c1a3571-1f3e-46f1-8361-0c84686fc618-kube-api-access-bk29k\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:36:49.399248 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.399060 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c1a3571-1f3e-46f1-8361-0c84686fc618-istio-token\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:36:49.892953 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.892919 2543 generic.go:356] "Generic (PLEG): container finished" podID="6c1a3571-1f3e-46f1-8361-0c84686fc618" containerID="3e1ab5dcaa4c9815da3f44c7e17aface4c7ef141f711710d1b66fa2915b5680c" exitCode=0 Jul 29 01:36:49.893188 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.892969 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" Jul 29 01:36:49.893188 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.893011 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" event={"ID":"6c1a3571-1f3e-46f1-8361-0c84686fc618","Type":"ContainerDied","Data":"3e1ab5dcaa4c9815da3f44c7e17aface4c7ef141f711710d1b66fa2915b5680c"} Jul 29 01:36:49.893188 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.893074 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb" event={"ID":"6c1a3571-1f3e-46f1-8361-0c84686fc618","Type":"ContainerDied","Data":"ec516735d573fba1f466f722589d2ca7e8f3a0bddc4560f60ae65d58d4c08d6c"} Jul 29 01:36:49.893188 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.893104 2543 scope.go:117] "RemoveContainer" containerID="3e1ab5dcaa4c9815da3f44c7e17aface4c7ef141f711710d1b66fa2915b5680c" Jul 29 01:36:49.902058 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.902038 2543 scope.go:117] "RemoveContainer" containerID="415b64f05add9056b45610c067363a0a7ca4c48d029ce5f92da908b5a0c9a823" Jul 29 01:36:49.909795 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.909778 2543 scope.go:117] "RemoveContainer" containerID="3e1ab5dcaa4c9815da3f44c7e17aface4c7ef141f711710d1b66fa2915b5680c" Jul 29 01:36:49.910077 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:36:49.910057 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e1ab5dcaa4c9815da3f44c7e17aface4c7ef141f711710d1b66fa2915b5680c\": container with ID starting with 3e1ab5dcaa4c9815da3f44c7e17aface4c7ef141f711710d1b66fa2915b5680c not found: ID does not exist" containerID="3e1ab5dcaa4c9815da3f44c7e17aface4c7ef141f711710d1b66fa2915b5680c" Jul 29 01:36:49.910142 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.910084 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e1ab5dcaa4c9815da3f44c7e17aface4c7ef141f711710d1b66fa2915b5680c"} err="failed to get container status \"3e1ab5dcaa4c9815da3f44c7e17aface4c7ef141f711710d1b66fa2915b5680c\": rpc error: code = NotFound desc = could not find container \"3e1ab5dcaa4c9815da3f44c7e17aface4c7ef141f711710d1b66fa2915b5680c\": container with ID starting with 3e1ab5dcaa4c9815da3f44c7e17aface4c7ef141f711710d1b66fa2915b5680c not found: ID does not exist" Jul 29 01:36:49.910142 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.910102 2543 scope.go:117] "RemoveContainer" containerID="415b64f05add9056b45610c067363a0a7ca4c48d029ce5f92da908b5a0c9a823" Jul 29 01:36:49.910341 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:36:49.910323 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415b64f05add9056b45610c067363a0a7ca4c48d029ce5f92da908b5a0c9a823\": container with ID starting with 415b64f05add9056b45610c067363a0a7ca4c48d029ce5f92da908b5a0c9a823 not found: ID does not exist" containerID="415b64f05add9056b45610c067363a0a7ca4c48d029ce5f92da908b5a0c9a823" Jul 29 01:36:49.910389 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.910348 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415b64f05add9056b45610c067363a0a7ca4c48d029ce5f92da908b5a0c9a823"} err="failed to get container status \"415b64f05add9056b45610c067363a0a7ca4c48d029ce5f92da908b5a0c9a823\": rpc error: code = NotFound desc = could not find container \"415b64f05add9056b45610c067363a0a7ca4c48d029ce5f92da908b5a0c9a823\": container with ID starting with 415b64f05add9056b45610c067363a0a7ca4c48d029ce5f92da908b5a0c9a823 not found: ID does not exist" Jul 29 01:36:49.914968 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.914948 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb"] Jul 29 01:36:49.918561 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:49.918541 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f5ec11-openshift-default-6f46b979f8-dtznb"] Jul 29 01:36:50.343246 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:36:50.343217 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c1a3571-1f3e-46f1-8361-0c84686fc618" path="/var/lib/kubelet/pods/6c1a3571-1f3e-46f1-8361-0c84686fc618/volumes" Jul 29 01:37:10.454333 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:10.454296 2543 scope.go:117] "RemoveContainer" containerID="5c79818a51d1d241e6ccc7147326b40790b7904d999b66bd306f4d3edc916de9" Jul 29 01:37:20.292608 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.292578 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd"] Jul 29 01:37:20.293083 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.292942 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0039f4e3-41e9-4963-9ba8-15e597d7d40b" containerName="istio-proxy" Jul 29 01:37:20.293083 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.292956 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0039f4e3-41e9-4963-9ba8-15e597d7d40b" containerName="istio-proxy" Jul 29 01:37:20.293083 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.292969 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1a3571-1f3e-46f1-8361-0c84686fc618" containerName="istio-proxy" Jul 29 01:37:20.293083 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.292974 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1a3571-1f3e-46f1-8361-0c84686fc618" containerName="istio-proxy" Jul 29 01:37:20.293083 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.292987 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0039f4e3-41e9-4963-9ba8-15e597d7d40b" containerName="istio-proxy" Jul 29 01:37:20.293083 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.292992 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0039f4e3-41e9-4963-9ba8-15e597d7d40b" containerName="istio-proxy" Jul 29 01:37:20.293083 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.293001 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef9c36e4-a301-44c4-9ebe-1a7bf8639d34" containerName="revoke-keys" Jul 29 01:37:20.293083 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.293006 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef9c36e4-a301-44c4-9ebe-1a7bf8639d34" containerName="revoke-keys" Jul 29 01:37:20.293083 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.293014 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1a3571-1f3e-46f1-8361-0c84686fc618" containerName="istio-proxy" Jul 29 01:37:20.293083 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.293020 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1a3571-1f3e-46f1-8361-0c84686fc618" containerName="istio-proxy" Jul 29 01:37:20.293083 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.293085 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef9c36e4-a301-44c4-9ebe-1a7bf8639d34" containerName="revoke-keys" Jul 29 01:37:20.293419 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.293096 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="0039f4e3-41e9-4963-9ba8-15e597d7d40b" containerName="istio-proxy" Jul 29 01:37:20.293419 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.293102 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c1a3571-1f3e-46f1-8361-0c84686fc618" containerName="istio-proxy" Jul 29 01:37:20.293419 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.293109 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c1a3571-1f3e-46f1-8361-0c84686fc618" containerName="istio-proxy" Jul 29 01:37:20.296151 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.296136 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" Jul 29 01:37:20.298696 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.298674 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 01:37:20.298807 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.298732 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 29 01:37:20.298895 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.298882 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4126fe-serving-cert\"" Jul 29 01:37:20.299613 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.299597 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-drtj4\"" Jul 29 01:37:20.307974 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.307949 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd"] Jul 29 01:37:20.365913 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.365881 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9zrh\" (UniqueName: \"kubernetes.io/projected/1199efce-c65f-4100-afd6-1ba1e3f25aff-kube-api-access-p9zrh\") pod \"maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd\" (UID: \"1199efce-c65f-4100-afd6-1ba1e3f25aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" Jul 29 01:37:20.366207 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.366179 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1199efce-c65f-4100-afd6-1ba1e3f25aff-openshift-service-ca\") pod \"maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd\" (UID: \"1199efce-c65f-4100-afd6-1ba1e3f25aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" Jul 29 01:37:20.366313 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.366229 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1199efce-c65f-4100-afd6-1ba1e3f25aff-maas-api-tls\") pod \"maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd\" (UID: \"1199efce-c65f-4100-afd6-1ba1e3f25aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" Jul 29 01:37:20.467204 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.467166 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9zrh\" (UniqueName: \"kubernetes.io/projected/1199efce-c65f-4100-afd6-1ba1e3f25aff-kube-api-access-p9zrh\") pod \"maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd\" (UID: \"1199efce-c65f-4100-afd6-1ba1e3f25aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" Jul 29 01:37:20.467381 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.467231 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1199efce-c65f-4100-afd6-1ba1e3f25aff-openshift-service-ca\") pod \"maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd\" (UID: \"1199efce-c65f-4100-afd6-1ba1e3f25aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" Jul 29 01:37:20.467381 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.467256 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1199efce-c65f-4100-afd6-1ba1e3f25aff-maas-api-tls\") pod \"maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd\" (UID: \"1199efce-c65f-4100-afd6-1ba1e3f25aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" Jul 29 01:37:20.467381 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:37:20.467370 2543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-serving-cert: secret "maas-api-e2e-api-b-4126fe-serving-cert" not found Jul 29 01:37:20.467486 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:37:20.467457 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1199efce-c65f-4100-afd6-1ba1e3f25aff-maas-api-tls podName:1199efce-c65f-4100-afd6-1ba1e3f25aff nodeName:}" failed. No retries permitted until 2026-07-29 01:37:20.967436896 +0000 UTC m=+2591.238081195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1199efce-c65f-4100-afd6-1ba1e3f25aff-maas-api-tls") pod "maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" (UID: "1199efce-c65f-4100-afd6-1ba1e3f25aff") : secret "maas-api-e2e-api-b-4126fe-serving-cert" not found Jul 29 01:37:20.467904 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.467883 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1199efce-c65f-4100-afd6-1ba1e3f25aff-openshift-service-ca\") pod \"maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd\" (UID: \"1199efce-c65f-4100-afd6-1ba1e3f25aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" Jul 29 01:37:20.483263 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.483227 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9zrh\" (UniqueName: \"kubernetes.io/projected/1199efce-c65f-4100-afd6-1ba1e3f25aff-kube-api-access-p9zrh\") pod \"maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd\" (UID: \"1199efce-c65f-4100-afd6-1ba1e3f25aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" Jul 29 01:37:20.971134 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.971096 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1199efce-c65f-4100-afd6-1ba1e3f25aff-maas-api-tls\") pod \"maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd\" (UID: \"1199efce-c65f-4100-afd6-1ba1e3f25aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" Jul 29 01:37:20.973904 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:20.973873 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1199efce-c65f-4100-afd6-1ba1e3f25aff-maas-api-tls\") pod \"maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd\" (UID: \"1199efce-c65f-4100-afd6-1ba1e3f25aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" Jul 29 01:37:21.127237 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:21.127203 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527"] Jul 29 01:37:21.127609 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:21.127596 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="0039f4e3-41e9-4963-9ba8-15e597d7d40b" containerName="istio-proxy" Jul 29 01:37:21.130776 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:21.130760 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527" Jul 29 01:37:21.133906 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:21.133886 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4126fe-dockercfg-4nv8p\"" Jul 29 01:37:21.134287 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:21.134267 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4126fe\"" Jul 29 01:37:21.142259 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:21.142236 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527"] Jul 29 01:37:21.172849 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:21.172805 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fae8dea-8385-47dd-8b15-14a3820b62e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527\" (UID: \"8fae8dea-8385-47dd-8b15-14a3820b62e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527" Jul 29 01:37:21.172992 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:21.172936 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsgfz\" (UniqueName: \"kubernetes.io/projected/8fae8dea-8385-47dd-8b15-14a3820b62e8-kube-api-access-jsgfz\") pod \"payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527\" (UID: \"8fae8dea-8385-47dd-8b15-14a3820b62e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527" Jul 29 01:37:21.206797 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:21.206764 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" Jul 29 01:37:21.275791 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:21.275751 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsgfz\" (UniqueName: \"kubernetes.io/projected/8fae8dea-8385-47dd-8b15-14a3820b62e8-kube-api-access-jsgfz\") pod \"payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527\" (UID: \"8fae8dea-8385-47dd-8b15-14a3820b62e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527" Jul 29 01:37:21.276501 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:21.276473 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fae8dea-8385-47dd-8b15-14a3820b62e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527\" (UID: \"8fae8dea-8385-47dd-8b15-14a3820b62e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527" Jul 29 01:37:21.277600 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:21.277477 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fae8dea-8385-47dd-8b15-14a3820b62e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527\" (UID: \"8fae8dea-8385-47dd-8b15-14a3820b62e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527" Jul 29 01:37:21.286541 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:21.286513 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsgfz\" (UniqueName: \"kubernetes.io/projected/8fae8dea-8385-47dd-8b15-14a3820b62e8-kube-api-access-jsgfz\") pod \"payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527\" (UID: \"8fae8dea-8385-47dd-8b15-14a3820b62e8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527" Jul 29 01:37:21.352503 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:21.352479 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd"] Jul 29 01:37:21.354491 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:37:21.354462 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1199efce_c65f_4100_afd6_1ba1e3f25aff.slice/crio-b711aceb228d3b5bd83f67bce9ffe6e1e37471ffdc81f656a63b5e1bcdaa5c5a WatchSource:0}: Error finding container b711aceb228d3b5bd83f67bce9ffe6e1e37471ffdc81f656a63b5e1bcdaa5c5a: Status 404 returned error can't find the container with id b711aceb228d3b5bd83f67bce9ffe6e1e37471ffdc81f656a63b5e1bcdaa5c5a Jul 29 01:37:21.355771 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:21.355752 2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 01:37:21.441933 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:21.441896 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527" Jul 29 01:37:21.776425 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:21.776400 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527"] Jul 29 01:37:21.778559 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:37:21.778533 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fae8dea_8385_47dd_8b15_14a3820b62e8.slice/crio-b8f340df615599ead399c179e106963b4c99e6e30d07f1a8c1fb447b23614e82 WatchSource:0}: Error finding container b8f340df615599ead399c179e106963b4c99e6e30d07f1a8c1fb447b23614e82: Status 404 returned error can't find the container with id b8f340df615599ead399c179e106963b4c99e6e30d07f1a8c1fb447b23614e82 Jul 29 01:37:22.042854 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:22.042753 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527" event={"ID":"8fae8dea-8385-47dd-8b15-14a3820b62e8","Type":"ContainerStarted","Data":"b8f340df615599ead399c179e106963b4c99e6e30d07f1a8c1fb447b23614e82"} Jul 29 01:37:22.044252 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:22.044227 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" event={"ID":"1199efce-c65f-4100-afd6-1ba1e3f25aff","Type":"ContainerStarted","Data":"b711aceb228d3b5bd83f67bce9ffe6e1e37471ffdc81f656a63b5e1bcdaa5c5a"} Jul 29 01:37:25.063439 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:25.063400 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527" event={"ID":"8fae8dea-8385-47dd-8b15-14a3820b62e8","Type":"ContainerStarted","Data":"0e7afeeb0890c063ac733815b7b1c77e50f5c12a9bfe44033ff058afe78dce6c"} Jul 29 01:37:25.063905 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:25.063548 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527" Jul 29 01:37:25.064891 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:25.064870 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" event={"ID":"1199efce-c65f-4100-afd6-1ba1e3f25aff","Type":"ContainerStarted","Data":"e6b456276bb61fa9406894425dd0a4a951f189436fad0e7c91ec3d4083400047"} Jul 29 01:37:25.065006 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:25.064991 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" Jul 29 01:37:25.078646 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:25.078605 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527" podStartSLOduration=1.864445075 podStartE2EDuration="4.078587525s" totalImagesPullingTime="2.21414245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:37:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:37:25.078254332 +0000 UTC m=+2595.348898641" watchObservedRunningTime="2026-07-29 01:37:25.078587525 +0000 UTC m=+2595.349231830" Jul 29 01:37:25.097715 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:25.097662 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" podStartSLOduration=2.461772859 podStartE2EDuration="5.097647105s" totalImagesPullingTime="2.635874246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:37:25.095531287 +0000 UTC m=+2595.366175616" watchObservedRunningTime="2026-07-29 01:37:25.097647105 +0000 UTC m=+2595.368291473" Jul 29 01:37:31.075174 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:31.075130 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" Jul 29 01:37:36.073079 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:36.072991 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527" Jul 29 01:37:48.036569 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:48.036536 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa4526-fvsp8"] Jul 29 01:37:48.039806 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:48.039789 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa4526-fvsp8" Jul 29 01:37:48.042084 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:48.042056 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjvns\"" Jul 29 01:37:48.048361 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:48.048337 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa4526-fvsp8"] Jul 29 01:37:48.208385 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:48.208350 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d31124e-a020-4839-a05c-4d4175e7b8b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-aa4526-fvsp8\" (UID: \"4d31124e-a020-4839-a05c-4d4175e7b8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa4526-fvsp8" Jul 29 01:37:48.309444 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:48.309355 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d31124e-a020-4839-a05c-4d4175e7b8b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-aa4526-fvsp8\" (UID: \"4d31124e-a020-4839-a05c-4d4175e7b8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa4526-fvsp8" Jul 29 01:37:48.310039 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:48.310019 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d31124e-a020-4839-a05c-4d4175e7b8b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-aa4526-fvsp8\" (UID: \"4d31124e-a020-4839-a05c-4d4175e7b8b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa4526-fvsp8" Jul 29 01:37:48.351207 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:48.351181 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa4526-fvsp8" Jul 29 01:37:48.476439 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:48.476406 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa4526-fvsp8"] Jul 29 01:37:48.479657 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:37:48.479627 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d31124e_a020_4839_a05c_4d4175e7b8b2.slice/crio-016c3d7566c52c2993878babbc0eea0f277c4695714b3f46a49571114de33fec WatchSource:0}: Error finding container 016c3d7566c52c2993878babbc0eea0f277c4695714b3f46a49571114de33fec: Status 404 returned error can't find the container with id 016c3d7566c52c2993878babbc0eea0f277c4695714b3f46a49571114de33fec Jul 29 01:37:49.182729 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:49.182700 2543 generic.go:356] "Generic (PLEG): container finished" podID="4d31124e-a020-4839-a05c-4d4175e7b8b2" containerID="8a62d05bfb8536f21cbd3270ed8ab75f0be71b3402fb9b9201174f21e1a7f90d" exitCode=0 Jul 29 01:37:49.183100 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:49.182790 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa4526-fvsp8" event={"ID":"4d31124e-a020-4839-a05c-4d4175e7b8b2","Type":"ContainerDied","Data":"8a62d05bfb8536f21cbd3270ed8ab75f0be71b3402fb9b9201174f21e1a7f90d"} Jul 29 01:37:49.183100 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:49.182877 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa4526-fvsp8" event={"ID":"4d31124e-a020-4839-a05c-4d4175e7b8b2","Type":"ContainerStarted","Data":"016c3d7566c52c2993878babbc0eea0f277c4695714b3f46a49571114de33fec"} Jul 29 01:37:50.317949 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:50.317924 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa4526-fvsp8" Jul 29 01:37:50.428037 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:50.428004 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d31124e-a020-4839-a05c-4d4175e7b8b2-maas-api-service-ca\") pod \"4d31124e-a020-4839-a05c-4d4175e7b8b2\" (UID: \"4d31124e-a020-4839-a05c-4d4175e7b8b2\") " Jul 29 01:37:50.428377 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:50.428352 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d31124e-a020-4839-a05c-4d4175e7b8b2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4d31124e-a020-4839-a05c-4d4175e7b8b2" (UID: "4d31124e-a020-4839-a05c-4d4175e7b8b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:37:50.528592 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:50.528519 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d31124e-a020-4839-a05c-4d4175e7b8b2-maas-api-service-ca\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:37:51.193187 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:51.193154 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa4526-fvsp8" Jul 29 01:37:51.193366 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:51.193161 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa4526-fvsp8" event={"ID":"4d31124e-a020-4839-a05c-4d4175e7b8b2","Type":"ContainerDied","Data":"016c3d7566c52c2993878babbc0eea0f277c4695714b3f46a49571114de33fec"} Jul 29 01:37:51.193366 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:37:51.193283 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="016c3d7566c52c2993878babbc0eea0f277c4695714b3f46a49571114de33fec" Jul 29 01:38:03.791131 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:03.791097 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa4526-fvsp8"] Jul 29 01:38:03.794219 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:03.794189 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aa4526-fvsp8"] Jul 29 01:38:04.343030 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:04.342997 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d31124e-a020-4839-a05c-4d4175e7b8b2" path="/var/lib/kubelet/pods/4d31124e-a020-4839-a05c-4d4175e7b8b2/volumes" Jul 29 01:38:10.482352 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:10.482318 2543 scope.go:117] "RemoveContainer" containerID="cd123f1f3b33637e09f7b6c2aa6029de0852afb777331d98e748fcdb1cdd23f9" Jul 29 01:38:10.788452 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:10.788358 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4126fe-dcxnx"] Jul 29 01:38:10.788806 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:10.788789 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d31124e-a020-4839-a05c-4d4175e7b8b2" containerName="revoke-keys" Jul 29 01:38:10.788920 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:10.788810 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d31124e-a020-4839-a05c-4d4175e7b8b2" containerName="revoke-keys" Jul 29 01:38:10.788920 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:10.788901 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d31124e-a020-4839-a05c-4d4175e7b8b2" containerName="revoke-keys" Jul 29 01:38:10.793877 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:10.793858 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4126fe-dcxnx" Jul 29 01:38:10.796108 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:10.796085 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjvns\"" Jul 29 01:38:10.798903 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:10.798880 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4126fe-dcxnx"] Jul 29 01:38:10.906027 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:10.905995 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8590fe7c-c3fc-4b27-9903-b95b1f3ef7d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4126fe-dcxnx\" (UID: \"8590fe7c-c3fc-4b27-9903-b95b1f3ef7d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4126fe-dcxnx" Jul 29 01:38:11.007070 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:11.007028 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8590fe7c-c3fc-4b27-9903-b95b1f3ef7d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4126fe-dcxnx\" (UID: \"8590fe7c-c3fc-4b27-9903-b95b1f3ef7d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4126fe-dcxnx" Jul 29 01:38:11.007646 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:11.007625 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8590fe7c-c3fc-4b27-9903-b95b1f3ef7d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4126fe-dcxnx\" (UID: \"8590fe7c-c3fc-4b27-9903-b95b1f3ef7d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4126fe-dcxnx" Jul 29 01:38:11.105210 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:11.105181 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4126fe-dcxnx" Jul 29 01:38:11.231893 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:11.231868 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4126fe-dcxnx"] Jul 29 01:38:11.234147 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:38:11.234119 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8590fe7c_c3fc_4b27_9903_b95b1f3ef7d9.slice/crio-fd511eaaf301ac8f8cceafd1bc029054d8ab5ecf555dde5e994386e17cc7dcc6 WatchSource:0}: Error finding container fd511eaaf301ac8f8cceafd1bc029054d8ab5ecf555dde5e994386e17cc7dcc6: Status 404 returned error can't find the container with id fd511eaaf301ac8f8cceafd1bc029054d8ab5ecf555dde5e994386e17cc7dcc6 Jul 29 01:38:11.292429 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:11.292403 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4126fe-dcxnx" event={"ID":"8590fe7c-c3fc-4b27-9903-b95b1f3ef7d9","Type":"ContainerStarted","Data":"fd511eaaf301ac8f8cceafd1bc029054d8ab5ecf555dde5e994386e17cc7dcc6"} Jul 29 01:38:12.299113 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:12.299024 2543 generic.go:356] "Generic (PLEG): container finished" podID="8590fe7c-c3fc-4b27-9903-b95b1f3ef7d9" containerID="f0082cdc48ddbf80d6ec300f90419c333e8d4a0fda565f73b9ab07cbeb41f9b5" exitCode=0 Jul 29 01:38:12.299113 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:12.299072 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4126fe-dcxnx" event={"ID":"8590fe7c-c3fc-4b27-9903-b95b1f3ef7d9","Type":"ContainerDied","Data":"f0082cdc48ddbf80d6ec300f90419c333e8d4a0fda565f73b9ab07cbeb41f9b5"} Jul 29 01:38:13.440870 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:13.440847 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4126fe-dcxnx" Jul 29 01:38:13.528037 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:13.528006 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8590fe7c-c3fc-4b27-9903-b95b1f3ef7d9-maas-api-service-ca\") pod \"8590fe7c-c3fc-4b27-9903-b95b1f3ef7d9\" (UID: \"8590fe7c-c3fc-4b27-9903-b95b1f3ef7d9\") " Jul 29 01:38:13.528384 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:13.528362 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8590fe7c-c3fc-4b27-9903-b95b1f3ef7d9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8590fe7c-c3fc-4b27-9903-b95b1f3ef7d9" (UID: "8590fe7c-c3fc-4b27-9903-b95b1f3ef7d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:38:13.628564 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:13.628535 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8590fe7c-c3fc-4b27-9903-b95b1f3ef7d9-maas-api-service-ca\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:38:14.311412 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:14.311371 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4126fe-dcxnx" event={"ID":"8590fe7c-c3fc-4b27-9903-b95b1f3ef7d9","Type":"ContainerDied","Data":"fd511eaaf301ac8f8cceafd1bc029054d8ab5ecf555dde5e994386e17cc7dcc6"} Jul 29 01:38:14.311412 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:14.311398 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4126fe-dcxnx" Jul 29 01:38:14.311630 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:14.311446 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd511eaaf301ac8f8cceafd1bc029054d8ab5ecf555dde5e994386e17cc7dcc6" Jul 29 01:38:26.227274 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.227237 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd"] Jul 29 01:38:26.227891 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.227523 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" podUID="1199efce-c65f-4100-afd6-1ba1e3f25aff" containerName="maas-api" containerID="cri-o://e6b456276bb61fa9406894425dd0a4a951f189436fad0e7c91ec3d4083400047" gracePeriod=30 Jul 29 01:38:26.293940 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.293904 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527"] Jul 29 01:38:26.294245 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.294187 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527" podUID="8fae8dea-8385-47dd-8b15-14a3820b62e8" containerName="payload-pre-processing" containerID="cri-o://0e7afeeb0890c063ac733815b7b1c77e50f5c12a9bfe44033ff058afe78dce6c" gracePeriod=30 Jul 29 01:38:26.380872 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.380840 2543 generic.go:356] "Generic (PLEG): container finished" podID="1199efce-c65f-4100-afd6-1ba1e3f25aff" containerID="e6b456276bb61fa9406894425dd0a4a951f189436fad0e7c91ec3d4083400047" exitCode=0 Jul 29 01:38:26.381045 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.380896 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" event={"ID":"1199efce-c65f-4100-afd6-1ba1e3f25aff","Type":"ContainerDied","Data":"e6b456276bb61fa9406894425dd0a4a951f189436fad0e7c91ec3d4083400047"} Jul 29 01:38:26.433615 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.433581 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4126fe-dcxnx"] Jul 29 01:38:26.436805 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.436772 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4126fe-dcxnx"] Jul 29 01:38:26.546701 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.546673 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" Jul 29 01:38:26.573587 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.573560 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527" Jul 29 01:38:26.640241 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.640200 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1199efce-c65f-4100-afd6-1ba1e3f25aff-openshift-service-ca\") pod \"1199efce-c65f-4100-afd6-1ba1e3f25aff\" (UID: \"1199efce-c65f-4100-afd6-1ba1e3f25aff\") " Jul 29 01:38:26.640241 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.640247 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1199efce-c65f-4100-afd6-1ba1e3f25aff-maas-api-tls\") pod \"1199efce-c65f-4100-afd6-1ba1e3f25aff\" (UID: \"1199efce-c65f-4100-afd6-1ba1e3f25aff\") " Jul 29 01:38:26.640496 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.640273 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fae8dea-8385-47dd-8b15-14a3820b62e8-plugins-config-volume\") pod \"8fae8dea-8385-47dd-8b15-14a3820b62e8\" (UID: \"8fae8dea-8385-47dd-8b15-14a3820b62e8\") " Jul 29 01:38:26.640496 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.640298 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsgfz\" (UniqueName: \"kubernetes.io/projected/8fae8dea-8385-47dd-8b15-14a3820b62e8-kube-api-access-jsgfz\") pod \"8fae8dea-8385-47dd-8b15-14a3820b62e8\" (UID: \"8fae8dea-8385-47dd-8b15-14a3820b62e8\") " Jul 29 01:38:26.640496 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.640349 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9zrh\" (UniqueName: \"kubernetes.io/projected/1199efce-c65f-4100-afd6-1ba1e3f25aff-kube-api-access-p9zrh\") pod \"1199efce-c65f-4100-afd6-1ba1e3f25aff\" (UID: \"1199efce-c65f-4100-afd6-1ba1e3f25aff\") " Jul 29 01:38:26.640694 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.640667 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1199efce-c65f-4100-afd6-1ba1e3f25aff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1199efce-c65f-4100-afd6-1ba1e3f25aff" (UID: "1199efce-c65f-4100-afd6-1ba1e3f25aff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:38:26.640765 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.640667 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fae8dea-8385-47dd-8b15-14a3820b62e8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8fae8dea-8385-47dd-8b15-14a3820b62e8" (UID: "8fae8dea-8385-47dd-8b15-14a3820b62e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:38:26.642489 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.642463 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fae8dea-8385-47dd-8b15-14a3820b62e8-kube-api-access-jsgfz" (OuterVolumeSpecName: "kube-api-access-jsgfz") pod "8fae8dea-8385-47dd-8b15-14a3820b62e8" (UID: "8fae8dea-8385-47dd-8b15-14a3820b62e8"). InnerVolumeSpecName "kube-api-access-jsgfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:38:26.642608 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.642547 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1199efce-c65f-4100-afd6-1ba1e3f25aff-kube-api-access-p9zrh" (OuterVolumeSpecName: "kube-api-access-p9zrh") pod "1199efce-c65f-4100-afd6-1ba1e3f25aff" (UID: "1199efce-c65f-4100-afd6-1ba1e3f25aff"). InnerVolumeSpecName "kube-api-access-p9zrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:38:26.642608 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.642581 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1199efce-c65f-4100-afd6-1ba1e3f25aff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1199efce-c65f-4100-afd6-1ba1e3f25aff" (UID: "1199efce-c65f-4100-afd6-1ba1e3f25aff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 01:38:26.741649 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.741618 2543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1199efce-c65f-4100-afd6-1ba1e3f25aff-openshift-service-ca\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:38:26.741649 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.741644 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1199efce-c65f-4100-afd6-1ba1e3f25aff-maas-api-tls\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:38:26.741649 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.741654 2543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fae8dea-8385-47dd-8b15-14a3820b62e8-plugins-config-volume\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:38:26.741904 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.741664 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsgfz\" (UniqueName: \"kubernetes.io/projected/8fae8dea-8385-47dd-8b15-14a3820b62e8-kube-api-access-jsgfz\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:38:26.741904 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:26.741673 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9zrh\" (UniqueName: \"kubernetes.io/projected/1199efce-c65f-4100-afd6-1ba1e3f25aff-kube-api-access-p9zrh\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:38:27.387775 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:27.387744 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" Jul 29 01:38:27.388237 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:27.387783 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd" event={"ID":"1199efce-c65f-4100-afd6-1ba1e3f25aff","Type":"ContainerDied","Data":"b711aceb228d3b5bd83f67bce9ffe6e1e37471ffdc81f656a63b5e1bcdaa5c5a"} Jul 29 01:38:27.388237 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:27.387860 2543 scope.go:117] "RemoveContainer" containerID="e6b456276bb61fa9406894425dd0a4a951f189436fad0e7c91ec3d4083400047" Jul 29 01:38:27.389220 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:27.389040 2543 generic.go:356] "Generic (PLEG): container finished" podID="8fae8dea-8385-47dd-8b15-14a3820b62e8" containerID="0e7afeeb0890c063ac733815b7b1c77e50f5c12a9bfe44033ff058afe78dce6c" exitCode=0 Jul 29 01:38:27.389220 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:27.389070 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527" event={"ID":"8fae8dea-8385-47dd-8b15-14a3820b62e8","Type":"ContainerDied","Data":"0e7afeeb0890c063ac733815b7b1c77e50f5c12a9bfe44033ff058afe78dce6c"} Jul 29 01:38:27.389220 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:27.389076 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527" Jul 29 01:38:27.389220 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:27.389109 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527" event={"ID":"8fae8dea-8385-47dd-8b15-14a3820b62e8","Type":"ContainerDied","Data":"b8f340df615599ead399c179e106963b4c99e6e30d07f1a8c1fb447b23614e82"} Jul 29 01:38:27.397014 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:27.396893 2543 scope.go:117] "RemoveContainer" containerID="0e7afeeb0890c063ac733815b7b1c77e50f5c12a9bfe44033ff058afe78dce6c" Jul 29 01:38:27.404675 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:27.404655 2543 scope.go:117] "RemoveContainer" containerID="0e7afeeb0890c063ac733815b7b1c77e50f5c12a9bfe44033ff058afe78dce6c" Jul 29 01:38:27.404930 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:38:27.404912 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e7afeeb0890c063ac733815b7b1c77e50f5c12a9bfe44033ff058afe78dce6c\": container with ID starting with 0e7afeeb0890c063ac733815b7b1c77e50f5c12a9bfe44033ff058afe78dce6c not found: ID does not exist" containerID="0e7afeeb0890c063ac733815b7b1c77e50f5c12a9bfe44033ff058afe78dce6c" Jul 29 01:38:27.404999 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:27.404944 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e7afeeb0890c063ac733815b7b1c77e50f5c12a9bfe44033ff058afe78dce6c"} err="failed to get container status \"0e7afeeb0890c063ac733815b7b1c77e50f5c12a9bfe44033ff058afe78dce6c\": rpc error: code = NotFound desc = could not find container \"0e7afeeb0890c063ac733815b7b1c77e50f5c12a9bfe44033ff058afe78dce6c\": container with ID starting with 0e7afeeb0890c063ac733815b7b1c77e50f5c12a9bfe44033ff058afe78dce6c not found: ID does not exist" Jul 29 01:38:27.414091 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:27.414067 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527"] Jul 29 01:38:27.422136 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:27.422117 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4126fe-5fc6f8995b-p8527"] Jul 29 01:38:27.434131 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:27.434111 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd"] Jul 29 01:38:27.437882 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:27.437864 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4126fe-86f64b77f5-prvnd"] Jul 29 01:38:28.343574 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:28.343538 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1199efce-c65f-4100-afd6-1ba1e3f25aff" path="/var/lib/kubelet/pods/1199efce-c65f-4100-afd6-1ba1e3f25aff/volumes" Jul 29 01:38:28.343895 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:28.343882 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8590fe7c-c3fc-4b27-9903-b95b1f3ef7d9" path="/var/lib/kubelet/pods/8590fe7c-c3fc-4b27-9903-b95b1f3ef7d9/volumes" Jul 29 01:38:28.344200 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:28.344188 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fae8dea-8385-47dd-8b15-14a3820b62e8" path="/var/lib/kubelet/pods/8fae8dea-8385-47dd-8b15-14a3820b62e8/volumes" Jul 29 01:38:33.551581 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.551549 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk"] Jul 29 01:38:33.552102 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.552003 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8590fe7c-c3fc-4b27-9903-b95b1f3ef7d9" containerName="revoke-keys" Jul 29 01:38:33.552102 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.552022 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8590fe7c-c3fc-4b27-9903-b95b1f3ef7d9" containerName="revoke-keys" Jul 29 01:38:33.552102 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.552045 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1199efce-c65f-4100-afd6-1ba1e3f25aff" containerName="maas-api" Jul 29 01:38:33.552102 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.552052 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="1199efce-c65f-4100-afd6-1ba1e3f25aff" containerName="maas-api" Jul 29 01:38:33.552102 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.552077 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fae8dea-8385-47dd-8b15-14a3820b62e8" containerName="payload-pre-processing" Jul 29 01:38:33.552102 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.552087 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fae8dea-8385-47dd-8b15-14a3820b62e8" containerName="payload-pre-processing" Jul 29 01:38:33.552347 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.552173 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="1199efce-c65f-4100-afd6-1ba1e3f25aff" containerName="maas-api" Jul 29 01:38:33.552347 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.552187 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="8590fe7c-c3fc-4b27-9903-b95b1f3ef7d9" containerName="revoke-keys" Jul 29 01:38:33.552347 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.552200 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fae8dea-8385-47dd-8b15-14a3820b62e8" containerName="payload-pre-processing" Jul 29 01:38:33.556516 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.556497 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.558916 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.558892 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-5ea1e0-openshift-default-dockercfg-58rcv\"" Jul 29 01:38:33.565526 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.565499 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk"] Jul 29 01:38:33.703711 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.703675 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-token\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.703711 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.703716 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-credential-socket\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.703950 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.703748 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93dc3d48-5755-40a3-9d66-a501ef6144ab-istiod-ca-cert\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.703950 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.703785 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-workload-certs\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.703950 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.703855 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-podinfo\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.703950 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.703887 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-workload-socket\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.704085 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.703947 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-envoy\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.704085 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.704000 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-data\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.704085 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.704026 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwx6d\" (UniqueName: \"kubernetes.io/projected/93dc3d48-5755-40a3-9d66-a501ef6144ab-kube-api-access-mwx6d\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.804509 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.804420 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwx6d\" (UniqueName: \"kubernetes.io/projected/93dc3d48-5755-40a3-9d66-a501ef6144ab-kube-api-access-mwx6d\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.804509 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.804475 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-token\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.804509 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.804500 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-credential-socket\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.804783 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.804520 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93dc3d48-5755-40a3-9d66-a501ef6144ab-istiod-ca-cert\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.804783 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.804541 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-workload-certs\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.804783 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.804578 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-podinfo\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.804783 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.804612 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-workload-socket\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.804783 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.804636 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-envoy\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.804783 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.804673 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-data\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.805123 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.805006 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-credential-socket\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.805123 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.805052 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-workload-certs\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.805232 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.805147 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-data\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.805337 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.805300 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-workload-socket\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.805457 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.805432 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93dc3d48-5755-40a3-9d66-a501ef6144ab-istiod-ca-cert\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.807285 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.807258 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-podinfo\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.807453 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.807437 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-envoy\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.815370 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.815347 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-token\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.815811 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.815789 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwx6d\" (UniqueName: \"kubernetes.io/projected/93dc3d48-5755-40a3-9d66-a501ef6144ab-kube-api-access-mwx6d\") pod \"e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:33.867558 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:33.867521 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:34.026381 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:34.026356 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk"] Jul 29 01:38:34.028997 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:38:34.028969 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93dc3d48_5755_40a3_9d66_a501ef6144ab.slice/crio-ffb6064512ea0eb9bb4957c2a3d16006313356e0bcf79f78075105caa9783f02 WatchSource:0}: Error finding container ffb6064512ea0eb9bb4957c2a3d16006313356e0bcf79f78075105caa9783f02: Status 404 returned error can't find the container with id ffb6064512ea0eb9bb4957c2a3d16006313356e0bcf79f78075105caa9783f02 Jul 29 01:38:34.031005 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:34.030974 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:38:34.031108 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:34.031037 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:38:34.031108 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:34.031066 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:38:34.427712 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:34.427676 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" event={"ID":"93dc3d48-5755-40a3-9d66-a501ef6144ab","Type":"ContainerStarted","Data":"21879f7446aa5be80d66c1c37068f694d49a28bc8fda159529978231e19d2ecf"} Jul 29 01:38:34.427957 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:34.427739 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" event={"ID":"93dc3d48-5755-40a3-9d66-a501ef6144ab","Type":"ContainerStarted","Data":"ffb6064512ea0eb9bb4957c2a3d16006313356e0bcf79f78075105caa9783f02"} Jul 29 01:38:34.451502 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:34.451439 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" podStartSLOduration=1.451422159 podStartE2EDuration="1.451422159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:38:34.449155803 +0000 UTC m=+2664.719800109" watchObservedRunningTime="2026-07-29 01:38:34.451422159 +0000 UTC m=+2664.722066466" Jul 29 01:38:34.868124 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:34.868090 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:35.873616 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:35.873588 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:36.441180 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:36.441148 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:36.442267 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:36.442248 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:36.513501 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:36.513468 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl"] Jul 29 01:38:36.517056 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:36.517025 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" Jul 29 01:38:36.519590 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:36.519565 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-5ea1e0-serving-cert\"" Jul 29 01:38:36.519894 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:36.519873 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-drtj4\"" Jul 29 01:38:36.520039 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:36.519917 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 01:38:36.520039 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:36.519969 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 29 01:38:36.530746 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:36.530727 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl"] Jul 29 01:38:36.630281 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:36.630240 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgcwt\" (UniqueName: \"kubernetes.io/projected/24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1-kube-api-access-kgcwt\") pod \"maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl\" (UID: \"24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" Jul 29 01:38:36.630540 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:36.630390 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1-maas-api-tls\") pod \"maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl\" (UID: \"24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" Jul 29 01:38:36.630540 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:36.630488 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl\" (UID: \"24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" Jul 29 01:38:36.731715 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:36.731623 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl\" (UID: \"24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" Jul 29 01:38:36.731715 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:36.731685 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgcwt\" (UniqueName: \"kubernetes.io/projected/24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1-kube-api-access-kgcwt\") pod \"maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl\" (UID: \"24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" Jul 29 01:38:36.731946 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:36.731739 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1-maas-api-tls\") pod \"maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl\" (UID: \"24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" Jul 29 01:38:36.731946 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:38:36.731854 2543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-serving-cert: secret "maas-api-e2e-shared-a-5ea1e0-serving-cert" not found Jul 29 01:38:36.731946 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:38:36.731910 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1-maas-api-tls podName:24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1 nodeName:}" failed. No retries permitted until 2026-07-29 01:38:37.231893207 +0000 UTC m=+2667.502537514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1-maas-api-tls") pod "maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" (UID: "24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1") : secret "maas-api-e2e-shared-a-5ea1e0-serving-cert" not found Jul 29 01:38:36.732279 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:36.732260 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl\" (UID: \"24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" Jul 29 01:38:36.747097 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:36.747067 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgcwt\" (UniqueName: \"kubernetes.io/projected/24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1-kube-api-access-kgcwt\") pod \"maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl\" (UID: \"24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" Jul 29 01:38:37.236108 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:37.236073 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1-maas-api-tls\") pod \"maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl\" (UID: \"24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" Jul 29 01:38:37.238450 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:37.238431 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1-maas-api-tls\") pod \"maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl\" (UID: \"24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" Jul 29 01:38:37.340962 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:37.340923 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw"] Jul 29 01:38:37.344417 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:37.344395 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw" Jul 29 01:38:37.346675 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:37.346654 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-5ea1e0\"" Jul 29 01:38:37.353140 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:37.353119 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw"] Jul 29 01:38:37.429152 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:37.429115 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" Jul 29 01:38:37.438346 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:37.438321 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb62aecb-6dab-41b6-a34f-1998e886ac98-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw\" (UID: \"eb62aecb-6dab-41b6-a34f-1998e886ac98\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw" Jul 29 01:38:37.438474 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:37.438391 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snhch\" (UniqueName: \"kubernetes.io/projected/eb62aecb-6dab-41b6-a34f-1998e886ac98-kube-api-access-snhch\") pod \"payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw\" (UID: \"eb62aecb-6dab-41b6-a34f-1998e886ac98\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw" Jul 29 01:38:37.539563 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:37.539527 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snhch\" (UniqueName: \"kubernetes.io/projected/eb62aecb-6dab-41b6-a34f-1998e886ac98-kube-api-access-snhch\") pod \"payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw\" (UID: \"eb62aecb-6dab-41b6-a34f-1998e886ac98\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw" Jul 29 01:38:37.539867 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:37.539837 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb62aecb-6dab-41b6-a34f-1998e886ac98-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw\" (UID: \"eb62aecb-6dab-41b6-a34f-1998e886ac98\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw" Jul 29 01:38:37.540762 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:37.540734 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb62aecb-6dab-41b6-a34f-1998e886ac98-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw\" (UID: \"eb62aecb-6dab-41b6-a34f-1998e886ac98\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw" Jul 29 01:38:37.559600 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:37.559569 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snhch\" (UniqueName: \"kubernetes.io/projected/eb62aecb-6dab-41b6-a34f-1998e886ac98-kube-api-access-snhch\") pod \"payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw\" (UID: \"eb62aecb-6dab-41b6-a34f-1998e886ac98\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw" Jul 29 01:38:37.585708 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:37.585678 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl"] Jul 29 01:38:37.587550 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:38:37.587522 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c4f5c6_2bd1_4fd6_9aec_4c8b7aac17f1.slice/crio-4caad00e579b6b719e872bf2b69bea417bd00594eaca0d115d2086141333d07d WatchSource:0}: Error finding container 4caad00e579b6b719e872bf2b69bea417bd00594eaca0d115d2086141333d07d: Status 404 returned error can't find the container with id 4caad00e579b6b719e872bf2b69bea417bd00594eaca0d115d2086141333d07d Jul 29 01:38:37.656718 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:37.656685 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw" Jul 29 01:38:37.792932 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:37.792900 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw"] Jul 29 01:38:37.795379 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:38:37.795346 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb62aecb_6dab_41b6_a34f_1998e886ac98.slice/crio-4035ff2d2bdccd3fbf1b297319e1276cf78d6657264dab6a3daf79676af0fcb0 WatchSource:0}: Error finding container 4035ff2d2bdccd3fbf1b297319e1276cf78d6657264dab6a3daf79676af0fcb0: Status 404 returned error can't find the container with id 4035ff2d2bdccd3fbf1b297319e1276cf78d6657264dab6a3daf79676af0fcb0 Jul 29 01:38:38.452229 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:38.452188 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw" event={"ID":"eb62aecb-6dab-41b6-a34f-1998e886ac98","Type":"ContainerStarted","Data":"c0e36400a3e3a7f5cdda54edbe8b14225a83d86f4feb72e8f11ec735a2ee0d33"} Jul 29 01:38:38.452694 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:38.452248 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw" event={"ID":"eb62aecb-6dab-41b6-a34f-1998e886ac98","Type":"ContainerStarted","Data":"4035ff2d2bdccd3fbf1b297319e1276cf78d6657264dab6a3daf79676af0fcb0"} Jul 29 01:38:38.452694 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:38.452275 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw" Jul 29 01:38:38.453753 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:38.453720 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" event={"ID":"24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1","Type":"ContainerStarted","Data":"cf1ead341346840c6b153aaa0783370e45bf078614b2e54de3f0b6e5417186d0"} Jul 29 01:38:38.453897 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:38.453773 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" event={"ID":"24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1","Type":"ContainerStarted","Data":"4caad00e579b6b719e872bf2b69bea417bd00594eaca0d115d2086141333d07d"} Jul 29 01:38:38.453968 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:38.453913 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" Jul 29 01:38:38.467503 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:38.467458 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw" podStartSLOduration=1.4674451290000001 podStartE2EDuration="1.467445129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:38:38.466803624 +0000 UTC m=+2668.737447931" watchObservedRunningTime="2026-07-29 01:38:38.467445129 +0000 UTC m=+2668.738089436" Jul 29 01:38:44.464177 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:44.464150 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" Jul 29 01:38:44.486726 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:44.486676 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" podStartSLOduration=8.149848969 podStartE2EDuration="8.486659261s" totalImagesPullingTime="336.810292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:38:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:38:38.48404898 +0000 UTC m=+2668.754693289" watchObservedRunningTime="2026-07-29 01:38:44.486659261 +0000 UTC m=+2674.757303567" Jul 29 01:38:49.460049 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:49.460010 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw" podUID="eb62aecb-6dab-41b6-a34f-1998e886ac98" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.57:9004: connect: connection refused" Jul 29 01:38:50.516639 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:50.516540 2543 generic.go:356] "Generic (PLEG): container finished" podID="93dc3d48-5755-40a3-9d66-a501ef6144ab" containerID="21879f7446aa5be80d66c1c37068f694d49a28bc8fda159529978231e19d2ecf" exitCode=0 Jul 29 01:38:50.516639 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:50.516617 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" event={"ID":"93dc3d48-5755-40a3-9d66-a501ef6144ab","Type":"ContainerDied","Data":"21879f7446aa5be80d66c1c37068f694d49a28bc8fda159529978231e19d2ecf"} Jul 29 01:38:50.517072 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:50.516992 2543 scope.go:117] "RemoveContainer" containerID="21879f7446aa5be80d66c1c37068f694d49a28bc8fda159529978231e19d2ecf" Jul 29 01:38:50.517970 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:50.517945 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:38:50.518040 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:50.518008 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:38:50.518076 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:50.518057 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 29 01:38:51.441473 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:51.441435 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:51.528503 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:51.528461 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" event={"ID":"93dc3d48-5755-40a3-9d66-a501ef6144ab","Type":"ContainerStarted","Data":"00be825c4e51c8e87edfb985c600d793fc8bbd88353a646689c814300b1398f5"} Jul 29 01:38:51.868010 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:51.867969 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:51.869256 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:51.869230 2543 patch_prober.go:28] interesting pod/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 29 01:38:51.869391 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:51.869284 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" podUID="93dc3d48-5755-40a3-9d66-a501ef6144ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 29 01:38:52.868221 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:52.868184 2543 patch_prober.go:28] interesting pod/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 29 01:38:52.868628 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:52.868245 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" podUID="93dc3d48-5755-40a3-9d66-a501ef6144ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 29 01:38:53.868377 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:53.868343 2543 patch_prober.go:28] interesting pod/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 29 01:38:53.868759 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:53.868392 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" podUID="93dc3d48-5755-40a3-9d66-a501ef6144ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 29 01:38:54.868743 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:54.868702 2543 patch_prober.go:28] interesting pod/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 29 01:38:54.869159 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:54.868777 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" podUID="93dc3d48-5755-40a3-9d66-a501ef6144ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 29 01:38:55.871957 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:55.871927 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:55.872355 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:55.872285 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:55.872883 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:55.872865 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:38:59.459716 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:38:59.459685 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw" Jul 29 01:39:05.783446 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:39:05.783359 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq"] Jul 29 01:39:05.786294 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:39:05.786274 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq" Jul 29 01:39:05.788582 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:39:05.788559 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-da474e-dockercfg-k2q2h\"" Jul 29 01:39:05.788881 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:39:05.788863 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-da474e\"" Jul 29 01:39:05.798129 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:39:05.798108 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq"] Jul 29 01:39:05.927120 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:39:05.927082 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csnpm\" (UniqueName: \"kubernetes.io/projected/9b2a5e68-4682-48d5-ac5c-d2e938a08419-kube-api-access-csnpm\") pod \"payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq\" (UID: \"9b2a5e68-4682-48d5-ac5c-d2e938a08419\") " pod="openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq" Jul 29 01:39:05.927120 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:39:05.927126 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2a5e68-4682-48d5-ac5c-d2e938a08419-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq\" (UID: \"9b2a5e68-4682-48d5-ac5c-d2e938a08419\") " pod="openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq" Jul 29 01:39:06.028273 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:39:06.028237 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2a5e68-4682-48d5-ac5c-d2e938a08419-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq\" (UID: \"9b2a5e68-4682-48d5-ac5c-d2e938a08419\") " pod="openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq" Jul 29 01:39:06.028452 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:39:06.028381 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csnpm\" (UniqueName: \"kubernetes.io/projected/9b2a5e68-4682-48d5-ac5c-d2e938a08419-kube-api-access-csnpm\") pod \"payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq\" (UID: \"9b2a5e68-4682-48d5-ac5c-d2e938a08419\") " pod="openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq" Jul 29 01:39:06.029058 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:39:06.029042 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2a5e68-4682-48d5-ac5c-d2e938a08419-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq\" (UID: \"9b2a5e68-4682-48d5-ac5c-d2e938a08419\") " pod="openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq" Jul 29 01:39:06.037028 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:39:06.036960 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csnpm\" (UniqueName: \"kubernetes.io/projected/9b2a5e68-4682-48d5-ac5c-d2e938a08419-kube-api-access-csnpm\") pod \"payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq\" (UID: \"9b2a5e68-4682-48d5-ac5c-d2e938a08419\") " pod="openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq" Jul 29 01:39:06.096654 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:39:06.096609 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq" Jul 29 01:39:06.231296 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:39:06.231268 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq"] Jul 29 01:39:06.233097 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:39:06.233070 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2a5e68_4682_48d5_ac5c_d2e938a08419.slice/crio-b5e1a80a700c2fbdf113edfbe0bb884ee75cea6dc1b21c50b6429b4c107eda37 WatchSource:0}: Error finding container b5e1a80a700c2fbdf113edfbe0bb884ee75cea6dc1b21c50b6429b4c107eda37: Status 404 returned error can't find the container with id b5e1a80a700c2fbdf113edfbe0bb884ee75cea6dc1b21c50b6429b4c107eda37 Jul 29 01:39:06.603764 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:39:06.603726 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq" event={"ID":"9b2a5e68-4682-48d5-ac5c-d2e938a08419","Type":"ContainerStarted","Data":"bf1059cd659cb8011103828cdf97b66bd16bc00d469efe93ea9b45142364ffbd"} Jul 29 01:39:06.603978 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:39:06.603779 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq" event={"ID":"9b2a5e68-4682-48d5-ac5c-d2e938a08419","Type":"ContainerStarted","Data":"b5e1a80a700c2fbdf113edfbe0bb884ee75cea6dc1b21c50b6429b4c107eda37"} Jul 29 01:39:06.603978 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:39:06.603873 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq" Jul 29 01:39:06.621201 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:39:06.621146 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq" podStartSLOduration=1.621130012 podStartE2EDuration="1.621130012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:39:06.62000616 +0000 UTC m=+2696.890650482" watchObservedRunningTime="2026-07-29 01:39:06.621130012 +0000 UTC m=+2696.891774319" Jul 29 01:39:10.730750 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:39:10.730723 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 01:39:11.057805 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:39:11.057774 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 01:39:17.611600 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:39:17.611572 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq" Jul 29 01:42:10.530538 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:42:10.530468 2543 scope.go:117] "RemoveContainer" containerID="ca8db452f198072e02b430f160e09dd6b4044ea1d2256ebb77fb24f65772eade" Jul 29 01:44:10.555632 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:44:10.555605 2543 scope.go:117] "RemoveContainer" containerID="8a62d05bfb8536f21cbd3270ed8ab75f0be71b3402fb9b9201174f21e1a7f90d" Jul 29 01:44:10.758109 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:44:10.758080 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 01:44:11.087907 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:44:11.087876 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 01:45:00.136806 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:00.136767 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ea1e0-29754825-dnz9n"] Jul 29 01:45:00.140242 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:00.140220 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754825-sddpw"] Jul 29 01:45:00.140366 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:00.140333 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ea1e0-29754825-dnz9n" Jul 29 01:45:00.142779 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:00.142753 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjvns\"" Jul 29 01:45:00.143603 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:00.143581 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754825-sddpw" Jul 29 01:45:00.148375 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:00.148300 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754825-sddpw"] Jul 29 01:45:00.151362 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:00.151329 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ea1e0-29754825-dnz9n"] Jul 29 01:45:00.154532 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:00.154507 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ea1e0-29754825-dnz9n" Jul 29 01:45:00.161340 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:00.161320 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754825-sddpw" Jul 29 01:45:00.519036 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:00.519012 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ea1e0-29754825-dnz9n"] Jul 29 01:45:00.521298 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:45:00.521270 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75ed66a_24d7_41bf_a620_a7c340bb8595.slice/crio-99d49d371ff99086549a18a7208875e135ea0fdf9ed6e2f1f3e9c347098f1567 WatchSource:0}: Error finding container 99d49d371ff99086549a18a7208875e135ea0fdf9ed6e2f1f3e9c347098f1567: Status 404 returned error can't find the container with id 99d49d371ff99086549a18a7208875e135ea0fdf9ed6e2f1f3e9c347098f1567 Jul 29 01:45:00.539382 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:00.539357 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754825-sddpw"] Jul 29 01:45:00.540767 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:45:00.540743 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48f5eca_498c_4403_8622_5cc7720b8be5.slice/crio-a90f0dbe5ac580b03fad99bbcb2d3c268d982579f8c28aad1feb5e446d55f2f2 WatchSource:0}: Error finding container a90f0dbe5ac580b03fad99bbcb2d3c268d982579f8c28aad1feb5e446d55f2f2: Status 404 returned error can't find the container with id a90f0dbe5ac580b03fad99bbcb2d3c268d982579f8c28aad1feb5e446d55f2f2 Jul 29 01:45:01.361976 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:01.361937 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ea1e0-29754825-dnz9n" event={"ID":"d75ed66a-24d7-41bf-a620-a7c340bb8595","Type":"ContainerStarted","Data":"4e6f6243b0488e65f5e9db4ec75392b6f4d648ded0499100ab6a09b780b097e1"} Jul 29 01:45:01.362443 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:01.361989 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ea1e0-29754825-dnz9n" event={"ID":"d75ed66a-24d7-41bf-a620-a7c340bb8595","Type":"ContainerStarted","Data":"99d49d371ff99086549a18a7208875e135ea0fdf9ed6e2f1f3e9c347098f1567"} Jul 29 01:45:01.363419 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:01.363383 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754825-sddpw" event={"ID":"e48f5eca-498c-4403-8622-5cc7720b8be5","Type":"ContainerStarted","Data":"7cce8a5976bfa769eacfc1de271a1beaf4a1bb50d58fd2210832825388575d24"} Jul 29 01:45:01.363559 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:01.363427 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754825-sddpw" event={"ID":"e48f5eca-498c-4403-8622-5cc7720b8be5","Type":"ContainerStarted","Data":"a90f0dbe5ac580b03fad99bbcb2d3c268d982579f8c28aad1feb5e446d55f2f2"} Jul 29 01:45:01.376100 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:01.376051 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ea1e0-29754825-dnz9n" podStartSLOduration=1.376037483 podStartE2EDuration="1.376037483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:45:01.375849805 +0000 UTC m=+3051.646494111" watchObservedRunningTime="2026-07-29 01:45:01.376037483 +0000 UTC m=+3051.646681790" Jul 29 01:45:01.389969 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:01.389923 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754825-sddpw" podStartSLOduration=1.389909476 podStartE2EDuration="1.389909476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:45:01.389509889 +0000 UTC m=+3051.660154197" watchObservedRunningTime="2026-07-29 01:45:01.389909476 +0000 UTC m=+3051.660553817" Jul 29 01:45:02.369717 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:02.369679 2543 generic.go:356] "Generic (PLEG): container finished" podID="d75ed66a-24d7-41bf-a620-a7c340bb8595" containerID="4e6f6243b0488e65f5e9db4ec75392b6f4d648ded0499100ab6a09b780b097e1" exitCode=0 Jul 29 01:45:02.370197 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:02.369775 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ea1e0-29754825-dnz9n" event={"ID":"d75ed66a-24d7-41bf-a620-a7c340bb8595","Type":"ContainerDied","Data":"4e6f6243b0488e65f5e9db4ec75392b6f4d648ded0499100ab6a09b780b097e1"} Jul 29 01:45:02.371172 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:02.371151 2543 generic.go:356] "Generic (PLEG): container finished" podID="e48f5eca-498c-4403-8622-5cc7720b8be5" containerID="7cce8a5976bfa769eacfc1de271a1beaf4a1bb50d58fd2210832825388575d24" exitCode=0 Jul 29 01:45:02.371235 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:02.371187 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754825-sddpw" event={"ID":"e48f5eca-498c-4403-8622-5cc7720b8be5","Type":"ContainerDied","Data":"7cce8a5976bfa769eacfc1de271a1beaf4a1bb50d58fd2210832825388575d24"} Jul 29 01:45:03.527884 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:03.527863 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ea1e0-29754825-dnz9n" Jul 29 01:45:03.530606 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:03.530588 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754825-sddpw" Jul 29 01:45:04.384702 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:04.384642 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ea1e0-29754825-dnz9n" Jul 29 01:45:04.384913 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:04.384637 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ea1e0-29754825-dnz9n" event={"ID":"d75ed66a-24d7-41bf-a620-a7c340bb8595","Type":"ContainerDied","Data":"99d49d371ff99086549a18a7208875e135ea0fdf9ed6e2f1f3e9c347098f1567"} Jul 29 01:45:04.384913 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:04.384758 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99d49d371ff99086549a18a7208875e135ea0fdf9ed6e2f1f3e9c347098f1567" Jul 29 01:45:04.386299 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:04.386276 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754825-sddpw" Jul 29 01:45:04.386415 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:04.386311 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754825-sddpw" event={"ID":"e48f5eca-498c-4403-8622-5cc7720b8be5","Type":"ContainerDied","Data":"a90f0dbe5ac580b03fad99bbcb2d3c268d982579f8c28aad1feb5e446d55f2f2"} Jul 29 01:45:04.386415 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:04.386346 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a90f0dbe5ac580b03fad99bbcb2d3c268d982579f8c28aad1feb5e446d55f2f2" Jul 29 01:45:10.575048 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:45:10.575018 2543 scope.go:117] "RemoveContainer" containerID="f0082cdc48ddbf80d6ec300f90419c333e8d4a0fda565f73b9ab07cbeb41f9b5" Jul 29 01:48:25.833766 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:25.833731 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq"] Jul 29 01:48:25.834421 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:25.834149 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d75ed66a-24d7-41bf-a620-a7c340bb8595" containerName="cleanup" Jul 29 01:48:25.834421 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:25.834163 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d75ed66a-24d7-41bf-a620-a7c340bb8595" containerName="cleanup" Jul 29 01:48:25.834421 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:25.834175 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e48f5eca-498c-4403-8622-5cc7720b8be5" containerName="cleanup" Jul 29 01:48:25.834421 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:25.834182 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="e48f5eca-498c-4403-8622-5cc7720b8be5" containerName="cleanup" Jul 29 01:48:25.834421 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:25.834246 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="d75ed66a-24d7-41bf-a620-a7c340bb8595" containerName="cleanup" Jul 29 01:48:25.834421 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:25.834258 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="e48f5eca-498c-4403-8622-5cc7720b8be5" containerName="cleanup" Jul 29 01:48:25.837368 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:25.837343 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq" Jul 29 01:48:25.839436 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:25.839418 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-295648-serving-cert\"" Jul 29 01:48:25.847096 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:25.847069 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq"] Jul 29 01:48:26.011396 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.011357 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bttm2\" (UniqueName: \"kubernetes.io/projected/cb63cf49-dd44-44e4-8564-3be4e2360251-kube-api-access-bttm2\") pod \"maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq\" (UID: \"cb63cf49-dd44-44e4-8564-3be4e2360251\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq" Jul 29 01:48:26.011396 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.011403 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb63cf49-dd44-44e4-8564-3be4e2360251-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq\" (UID: \"cb63cf49-dd44-44e4-8564-3be4e2360251\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq" Jul 29 01:48:26.011612 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.011465 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb63cf49-dd44-44e4-8564-3be4e2360251-maas-api-tls\") pod \"maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq\" (UID: \"cb63cf49-dd44-44e4-8564-3be4e2360251\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq" Jul 29 01:48:26.112473 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.112377 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb63cf49-dd44-44e4-8564-3be4e2360251-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq\" (UID: \"cb63cf49-dd44-44e4-8564-3be4e2360251\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq" Jul 29 01:48:26.112473 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.112445 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb63cf49-dd44-44e4-8564-3be4e2360251-maas-api-tls\") pod \"maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq\" (UID: \"cb63cf49-dd44-44e4-8564-3be4e2360251\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq" Jul 29 01:48:26.112706 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.112550 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bttm2\" (UniqueName: \"kubernetes.io/projected/cb63cf49-dd44-44e4-8564-3be4e2360251-kube-api-access-bttm2\") pod \"maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq\" (UID: \"cb63cf49-dd44-44e4-8564-3be4e2360251\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq" Jul 29 01:48:26.113061 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.113039 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb63cf49-dd44-44e4-8564-3be4e2360251-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq\" (UID: \"cb63cf49-dd44-44e4-8564-3be4e2360251\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq" Jul 29 01:48:26.114827 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.114801 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb63cf49-dd44-44e4-8564-3be4e2360251-maas-api-tls\") pod \"maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq\" (UID: \"cb63cf49-dd44-44e4-8564-3be4e2360251\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq" Jul 29 01:48:26.122340 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.122315 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bttm2\" (UniqueName: \"kubernetes.io/projected/cb63cf49-dd44-44e4-8564-3be4e2360251-kube-api-access-bttm2\") pod \"maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq\" (UID: \"cb63cf49-dd44-44e4-8564-3be4e2360251\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq" Jul 29 01:48:26.148071 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.148040 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq" Jul 29 01:48:26.283857 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.282736 2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 01:48:26.286348 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.286326 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq"] Jul 29 01:48:26.383842 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.383751 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq" event={"ID":"cb63cf49-dd44-44e4-8564-3be4e2360251","Type":"ContainerStarted","Data":"bcefb6748477b17e41297ae3b795f23f23a8dccd73b1760b3f8acf69aed5bbd8"} Jul 29 01:48:26.617581 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.617543 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw"] Jul 29 01:48:26.622059 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.622039 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw" Jul 29 01:48:26.624129 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.624108 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-295648\"" Jul 29 01:48:26.631476 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.631456 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw"] Jul 29 01:48:26.650389 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.650324 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957"] Jul 29 01:48:26.653735 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.653718 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957" Jul 29 01:48:26.655874 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.655855 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-295648-dockercfg-vrz6d\"" Jul 29 01:48:26.666386 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.666364 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957"] Jul 29 01:48:26.718697 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.718673 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a31040a-c8a7-45ac-b508-5cac456360a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw\" (UID: \"4a31040a-c8a7-45ac-b508-5cac456360a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw" Jul 29 01:48:26.718793 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.718706 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b25p2\" (UniqueName: \"kubernetes.io/projected/4a31040a-c8a7-45ac-b508-5cac456360a6-kube-api-access-b25p2\") pod \"payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw\" (UID: \"4a31040a-c8a7-45ac-b508-5cac456360a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw" Jul 29 01:48:26.820166 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.820126 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rkkj\" (UniqueName: \"kubernetes.io/projected/3f6cf9cc-204c-4ac0-8889-81eb506a1375-kube-api-access-2rkkj\") pod \"payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957\" (UID: \"3f6cf9cc-204c-4ac0-8889-81eb506a1375\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957" Jul 29 01:48:26.820366 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.820173 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a31040a-c8a7-45ac-b508-5cac456360a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw\" (UID: \"4a31040a-c8a7-45ac-b508-5cac456360a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw" Jul 29 01:48:26.820366 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.820236 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b25p2\" (UniqueName: \"kubernetes.io/projected/4a31040a-c8a7-45ac-b508-5cac456360a6-kube-api-access-b25p2\") pod \"payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw\" (UID: \"4a31040a-c8a7-45ac-b508-5cac456360a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw" Jul 29 01:48:26.820366 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.820331 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f6cf9cc-204c-4ac0-8889-81eb506a1375-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957\" (UID: \"3f6cf9cc-204c-4ac0-8889-81eb506a1375\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957" Jul 29 01:48:26.820939 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.820918 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a31040a-c8a7-45ac-b508-5cac456360a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw\" (UID: \"4a31040a-c8a7-45ac-b508-5cac456360a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw" Jul 29 01:48:26.830413 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.830393 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b25p2\" (UniqueName: \"kubernetes.io/projected/4a31040a-c8a7-45ac-b508-5cac456360a6-kube-api-access-b25p2\") pod \"payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw\" (UID: \"4a31040a-c8a7-45ac-b508-5cac456360a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw" Jul 29 01:48:26.921265 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.921181 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f6cf9cc-204c-4ac0-8889-81eb506a1375-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957\" (UID: \"3f6cf9cc-204c-4ac0-8889-81eb506a1375\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957" Jul 29 01:48:26.921621 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.921276 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rkkj\" (UniqueName: \"kubernetes.io/projected/3f6cf9cc-204c-4ac0-8889-81eb506a1375-kube-api-access-2rkkj\") pod \"payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957\" (UID: \"3f6cf9cc-204c-4ac0-8889-81eb506a1375\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957" Jul 29 01:48:26.921883 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.921865 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f6cf9cc-204c-4ac0-8889-81eb506a1375-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957\" (UID: \"3f6cf9cc-204c-4ac0-8889-81eb506a1375\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957" Jul 29 01:48:26.931983 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.931961 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rkkj\" (UniqueName: \"kubernetes.io/projected/3f6cf9cc-204c-4ac0-8889-81eb506a1375-kube-api-access-2rkkj\") pod \"payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957\" (UID: \"3f6cf9cc-204c-4ac0-8889-81eb506a1375\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957" Jul 29 01:48:26.933751 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.933733 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw" Jul 29 01:48:26.963852 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:26.963808 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957" Jul 29 01:48:27.078957 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:27.078790 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw"] Jul 29 01:48:27.081212 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:48:27.081181 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a31040a_c8a7_45ac_b508_5cac456360a6.slice/crio-607caeddce3dfd3d48163ea9c78c548c756e45ca73ed7ec01e0489194be16441 WatchSource:0}: Error finding container 607caeddce3dfd3d48163ea9c78c548c756e45ca73ed7ec01e0489194be16441: Status 404 returned error can't find the container with id 607caeddce3dfd3d48163ea9c78c548c756e45ca73ed7ec01e0489194be16441 Jul 29 01:48:27.113340 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:27.113170 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957"] Jul 29 01:48:27.119063 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:48:27.119033 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6cf9cc_204c_4ac0_8889_81eb506a1375.slice/crio-152b7ff0ed70b5d988a911b3515fd2d9cb5c79d0250dd77bf1aca442628f0a1a WatchSource:0}: Error finding container 152b7ff0ed70b5d988a911b3515fd2d9cb5c79d0250dd77bf1aca442628f0a1a: Status 404 returned error can't find the container with id 152b7ff0ed70b5d988a911b3515fd2d9cb5c79d0250dd77bf1aca442628f0a1a Jul 29 01:48:27.390786 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:27.390745 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957" event={"ID":"3f6cf9cc-204c-4ac0-8889-81eb506a1375","Type":"ContainerStarted","Data":"b30d3f9f8e7bbced5ca8358a0ee800d33de0efdca693afdae9052f7a6b658d7a"} Jul 29 01:48:27.391021 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:27.390804 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957" event={"ID":"3f6cf9cc-204c-4ac0-8889-81eb506a1375","Type":"ContainerStarted","Data":"152b7ff0ed70b5d988a911b3515fd2d9cb5c79d0250dd77bf1aca442628f0a1a"} Jul 29 01:48:27.391021 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:27.390990 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957" Jul 29 01:48:27.392693 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:27.392658 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq" event={"ID":"cb63cf49-dd44-44e4-8564-3be4e2360251","Type":"ContainerStarted","Data":"8be92792afb147805c5996d63b693b1dba27c9c9d614786332ef5d1bfb0e7791"} Jul 29 01:48:27.392874 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:27.392729 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq" Jul 29 01:48:27.394471 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:27.394448 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw" event={"ID":"4a31040a-c8a7-45ac-b508-5cac456360a6","Type":"ContainerStarted","Data":"924740bda7dab33258bf1601a57d66051451f07efbc68de118939b23d0579d4f"} Jul 29 01:48:27.394619 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:27.394489 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw" event={"ID":"4a31040a-c8a7-45ac-b508-5cac456360a6","Type":"ContainerStarted","Data":"607caeddce3dfd3d48163ea9c78c548c756e45ca73ed7ec01e0489194be16441"} Jul 29 01:48:27.394619 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:27.394514 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw" Jul 29 01:48:27.408776 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:27.408736 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957" podStartSLOduration=1.408724756 podStartE2EDuration="1.408724756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:48:27.406631355 +0000 UTC m=+3257.677275665" watchObservedRunningTime="2026-07-29 01:48:27.408724756 +0000 UTC m=+3257.679369064" Jul 29 01:48:27.424199 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:27.424127 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq" podStartSLOduration=2.019228224 podStartE2EDuration="2.42411546s" totalImagesPullingTime="404.887236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:48:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:48:27.423334032 +0000 UTC m=+3257.693978341" watchObservedRunningTime="2026-07-29 01:48:27.42411546 +0000 UTC m=+3257.694759768" Jul 29 01:48:27.439310 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:27.439270 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw" podStartSLOduration=1.4392598300000001 podStartE2EDuration="1.43925983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:48:27.437429036 +0000 UTC m=+3257.708073344" watchObservedRunningTime="2026-07-29 01:48:27.43925983 +0000 UTC m=+3257.709904136" Jul 29 01:48:33.408110 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:33.408081 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq" Jul 29 01:48:38.402278 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:38.402246 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957" Jul 29 01:48:38.402733 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:38.402318 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw" Jul 29 01:48:55.044906 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.044862 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k"] Jul 29 01:48:55.048589 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.048570 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" Jul 29 01:48:55.050682 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.050657 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-8230fd-serving-cert\"" Jul 29 01:48:55.059793 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.059770 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k"] Jul 29 01:48:55.171149 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.171107 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snxfm\" (UniqueName: \"kubernetes.io/projected/c283ffd5-04ec-4c64-8019-45da4b5aad0b-kube-api-access-snxfm\") pod \"maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k\" (UID: \"c283ffd5-04ec-4c64-8019-45da4b5aad0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" Jul 29 01:48:55.171327 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.171166 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c283ffd5-04ec-4c64-8019-45da4b5aad0b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k\" (UID: \"c283ffd5-04ec-4c64-8019-45da4b5aad0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" Jul 29 01:48:55.171327 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.171296 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c283ffd5-04ec-4c64-8019-45da4b5aad0b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k\" (UID: \"c283ffd5-04ec-4c64-8019-45da4b5aad0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" Jul 29 01:48:55.272175 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.272143 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c283ffd5-04ec-4c64-8019-45da4b5aad0b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k\" (UID: \"c283ffd5-04ec-4c64-8019-45da4b5aad0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" Jul 29 01:48:55.272331 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.272201 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snxfm\" (UniqueName: \"kubernetes.io/projected/c283ffd5-04ec-4c64-8019-45da4b5aad0b-kube-api-access-snxfm\") pod \"maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k\" (UID: \"c283ffd5-04ec-4c64-8019-45da4b5aad0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" Jul 29 01:48:55.272331 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.272241 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c283ffd5-04ec-4c64-8019-45da4b5aad0b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k\" (UID: \"c283ffd5-04ec-4c64-8019-45da4b5aad0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" Jul 29 01:48:55.272408 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:48:55.272354 2543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-serving-cert: secret "maas-api-e2e-ipp-b-8230fd-serving-cert" not found Jul 29 01:48:55.272459 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:48:55.272411 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c283ffd5-04ec-4c64-8019-45da4b5aad0b-maas-api-tls podName:c283ffd5-04ec-4c64-8019-45da4b5aad0b nodeName:}" failed. No retries permitted until 2026-07-29 01:48:55.772392652 +0000 UTC m=+3286.043036953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c283ffd5-04ec-4c64-8019-45da4b5aad0b-maas-api-tls") pod "maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" (UID: "c283ffd5-04ec-4c64-8019-45da4b5aad0b") : secret "maas-api-e2e-ipp-b-8230fd-serving-cert" not found Jul 29 01:48:55.272753 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.272732 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c283ffd5-04ec-4c64-8019-45da4b5aad0b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k\" (UID: \"c283ffd5-04ec-4c64-8019-45da4b5aad0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" Jul 29 01:48:55.280950 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.280930 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snxfm\" (UniqueName: \"kubernetes.io/projected/c283ffd5-04ec-4c64-8019-45da4b5aad0b-kube-api-access-snxfm\") pod \"maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k\" (UID: \"c283ffd5-04ec-4c64-8019-45da4b5aad0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" Jul 29 01:48:55.766457 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.766425 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph"] Jul 29 01:48:55.769994 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.769978 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph" Jul 29 01:48:55.772122 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.772096 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8230fd-dockercfg-xrzzq\"" Jul 29 01:48:55.772250 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.772124 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8230fd\"" Jul 29 01:48:55.777641 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.777614 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c283ffd5-04ec-4c64-8019-45da4b5aad0b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k\" (UID: \"c283ffd5-04ec-4c64-8019-45da4b5aad0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" Jul 29 01:48:55.777895 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:48:55.777872 2543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-serving-cert: secret "maas-api-e2e-ipp-b-8230fd-serving-cert" not found Jul 29 01:48:55.778025 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:48:55.778009 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c283ffd5-04ec-4c64-8019-45da4b5aad0b-maas-api-tls podName:c283ffd5-04ec-4c64-8019-45da4b5aad0b nodeName:}" failed. No retries permitted until 2026-07-29 01:48:56.77797306 +0000 UTC m=+3287.048617359 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c283ffd5-04ec-4c64-8019-45da4b5aad0b-maas-api-tls") pod "maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" (UID: "c283ffd5-04ec-4c64-8019-45da4b5aad0b") : secret "maas-api-e2e-ipp-b-8230fd-serving-cert" not found Jul 29 01:48:55.780076 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.780052 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph"] Jul 29 01:48:55.878982 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.878942 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b8tz\" (UniqueName: \"kubernetes.io/projected/9b0743f8-60ec-4f0f-a2a5-98ab5f206db7-kube-api-access-6b8tz\") pod \"payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph\" (UID: \"9b0743f8-60ec-4f0f-a2a5-98ab5f206db7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph" Jul 29 01:48:55.879187 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.878986 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0743f8-60ec-4f0f-a2a5-98ab5f206db7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph\" (UID: \"9b0743f8-60ec-4f0f-a2a5-98ab5f206db7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph" Jul 29 01:48:55.979916 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.979883 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b8tz\" (UniqueName: \"kubernetes.io/projected/9b0743f8-60ec-4f0f-a2a5-98ab5f206db7-kube-api-access-6b8tz\") pod \"payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph\" (UID: \"9b0743f8-60ec-4f0f-a2a5-98ab5f206db7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph" Jul 29 01:48:55.979916 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.979921 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0743f8-60ec-4f0f-a2a5-98ab5f206db7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph\" (UID: \"9b0743f8-60ec-4f0f-a2a5-98ab5f206db7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph" Jul 29 01:48:55.980599 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.980581 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0743f8-60ec-4f0f-a2a5-98ab5f206db7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph\" (UID: \"9b0743f8-60ec-4f0f-a2a5-98ab5f206db7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph" Jul 29 01:48:55.987030 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:55.987001 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b8tz\" (UniqueName: \"kubernetes.io/projected/9b0743f8-60ec-4f0f-a2a5-98ab5f206db7-kube-api-access-6b8tz\") pod \"payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph\" (UID: \"9b0743f8-60ec-4f0f-a2a5-98ab5f206db7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph" Jul 29 01:48:56.081345 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:56.081318 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph" Jul 29 01:48:56.212610 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:56.212584 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph"] Jul 29 01:48:56.214651 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:48:56.214626 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0743f8_60ec_4f0f_a2a5_98ab5f206db7.slice/crio-732bfcc331ebfce1b69d01c64908d527ef850de4899c44188505d987cbe63655 WatchSource:0}: Error finding container 732bfcc331ebfce1b69d01c64908d527ef850de4899c44188505d987cbe63655: Status 404 returned error can't find the container with id 732bfcc331ebfce1b69d01c64908d527ef850de4899c44188505d987cbe63655 Jul 29 01:48:56.547273 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:56.547182 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph" event={"ID":"9b0743f8-60ec-4f0f-a2a5-98ab5f206db7","Type":"ContainerStarted","Data":"6ca09207ffab2b200540d39e69dc808f89dead991a5f6c30093354e7d8e6020e"} Jul 29 01:48:56.547273 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:56.547235 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph" event={"ID":"9b0743f8-60ec-4f0f-a2a5-98ab5f206db7","Type":"ContainerStarted","Data":"732bfcc331ebfce1b69d01c64908d527ef850de4899c44188505d987cbe63655"} Jul 29 01:48:56.547273 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:56.547265 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph" Jul 29 01:48:56.563665 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:56.563618 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph" podStartSLOduration=1.563602639 podStartE2EDuration="1.563602639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:48:56.561763975 +0000 UTC m=+3286.832408271" watchObservedRunningTime="2026-07-29 01:48:56.563602639 +0000 UTC m=+3286.834246945" Jul 29 01:48:56.789674 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:56.789636 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c283ffd5-04ec-4c64-8019-45da4b5aad0b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k\" (UID: \"c283ffd5-04ec-4c64-8019-45da4b5aad0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" Jul 29 01:48:56.792063 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:56.792044 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c283ffd5-04ec-4c64-8019-45da4b5aad0b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k\" (UID: \"c283ffd5-04ec-4c64-8019-45da4b5aad0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" Jul 29 01:48:56.859743 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:56.859717 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" Jul 29 01:48:56.994110 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:56.994067 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k"] Jul 29 01:48:56.996554 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:48:56.996518 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc283ffd5_04ec_4c64_8019_45da4b5aad0b.slice/crio-2bce3a9ba55014076d68ac1e1e8dd13feaec8a4c2ccb6e529adaf1e1fbd6ac27 WatchSource:0}: Error finding container 2bce3a9ba55014076d68ac1e1e8dd13feaec8a4c2ccb6e529adaf1e1fbd6ac27: Status 404 returned error can't find the container with id 2bce3a9ba55014076d68ac1e1e8dd13feaec8a4c2ccb6e529adaf1e1fbd6ac27 Jul 29 01:48:57.552975 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:57.552937 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" event={"ID":"c283ffd5-04ec-4c64-8019-45da4b5aad0b","Type":"ContainerStarted","Data":"2bce3a9ba55014076d68ac1e1e8dd13feaec8a4c2ccb6e529adaf1e1fbd6ac27"} Jul 29 01:48:58.560206 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:58.560167 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" event={"ID":"c283ffd5-04ec-4c64-8019-45da4b5aad0b","Type":"ContainerStarted","Data":"0c52c331bc6d402d62005743c4dc2a04270f0b8f2677fc38252c2445434299a9"} Jul 29 01:48:58.560594 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:58.560266 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" Jul 29 01:48:58.578465 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:48:58.578412 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" podStartSLOduration=2.82775638 podStartE2EDuration="3.578397589s" totalImagesPullingTime="750.641209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:48:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:48:58.577393736 +0000 UTC m=+3288.848038044" watchObservedRunningTime="2026-07-29 01:48:58.578397589 +0000 UTC m=+3288.849041892" Jul 29 01:49:04.570501 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:49:04.570471 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" Jul 29 01:49:07.555990 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:49:07.555957 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph" Jul 29 01:49:10.787721 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:49:10.787693 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 01:49:11.125004 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:49:11.124979 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 01:49:26.834071 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:49:26.834040 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph_9b0743f8-60ec-4f0f-a2a5-98ab5f206db7/payload-processing/0.log" Jul 29 01:50:13.612316 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:13.612283 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957_3f6cf9cc-204c-4ac0-8889-81eb506a1375/payload-processing/0.log" Jul 29 01:50:13.839745 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:13.839715 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph_9b0743f8-60ec-4f0f-a2a5-98ab5f206db7/payload-processing/0.log" Jul 29 01:50:17.710210 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:17.710176 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4"] Jul 29 01:50:17.713953 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:17.713934 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" Jul 29 01:50:17.716156 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:17.716132 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-dc8b9a-serving-cert\"" Jul 29 01:50:17.725774 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:17.725751 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4"] Jul 29 01:50:17.848164 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:17.848124 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a6a9869-1068-4c80-8773-70ebf4f6d1e9-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4\" (UID: \"5a6a9869-1068-4c80-8773-70ebf4f6d1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" Jul 29 01:50:17.848354 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:17.848172 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a6a9869-1068-4c80-8773-70ebf4f6d1e9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4\" (UID: \"5a6a9869-1068-4c80-8773-70ebf4f6d1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" Jul 29 01:50:17.848354 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:17.848304 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhngv\" (UniqueName: \"kubernetes.io/projected/5a6a9869-1068-4c80-8773-70ebf4f6d1e9-kube-api-access-dhngv\") pod \"maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4\" (UID: \"5a6a9869-1068-4c80-8773-70ebf4f6d1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" Jul 29 01:50:17.949802 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:17.949764 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhngv\" (UniqueName: \"kubernetes.io/projected/5a6a9869-1068-4c80-8773-70ebf4f6d1e9-kube-api-access-dhngv\") pod \"maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4\" (UID: \"5a6a9869-1068-4c80-8773-70ebf4f6d1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" Jul 29 01:50:17.950018 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:17.949869 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a6a9869-1068-4c80-8773-70ebf4f6d1e9-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4\" (UID: \"5a6a9869-1068-4c80-8773-70ebf4f6d1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" Jul 29 01:50:17.950018 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:17.949911 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a6a9869-1068-4c80-8773-70ebf4f6d1e9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4\" (UID: \"5a6a9869-1068-4c80-8773-70ebf4f6d1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" Jul 29 01:50:17.950153 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:50:17.950037 2543 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-serving-cert: secret "maas-api-e2e-ipp-cleanup-dc8b9a-serving-cert" not found Jul 29 01:50:17.950153 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:50:17.950125 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a6a9869-1068-4c80-8773-70ebf4f6d1e9-maas-api-tls podName:5a6a9869-1068-4c80-8773-70ebf4f6d1e9 nodeName:}" failed. No retries permitted until 2026-07-29 01:50:18.450098462 +0000 UTC m=+3368.720742750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5a6a9869-1068-4c80-8773-70ebf4f6d1e9-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" (UID: "5a6a9869-1068-4c80-8773-70ebf4f6d1e9") : secret "maas-api-e2e-ipp-cleanup-dc8b9a-serving-cert" not found Jul 29 01:50:17.950525 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:17.950503 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a6a9869-1068-4c80-8773-70ebf4f6d1e9-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4\" (UID: \"5a6a9869-1068-4c80-8773-70ebf4f6d1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" Jul 29 01:50:17.964188 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:17.964118 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhngv\" (UniqueName: \"kubernetes.io/projected/5a6a9869-1068-4c80-8773-70ebf4f6d1e9-kube-api-access-dhngv\") pod \"maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4\" (UID: \"5a6a9869-1068-4c80-8773-70ebf4f6d1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" Jul 29 01:50:18.455391 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.455350 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a6a9869-1068-4c80-8773-70ebf4f6d1e9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4\" (UID: \"5a6a9869-1068-4c80-8773-70ebf4f6d1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" Jul 29 01:50:18.458030 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.457998 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a6a9869-1068-4c80-8773-70ebf4f6d1e9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4\" (UID: \"5a6a9869-1068-4c80-8773-70ebf4f6d1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" Jul 29 01:50:18.458774 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.458754 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz"] Jul 29 01:50:18.466031 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.465998 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz" Jul 29 01:50:18.468891 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.468856 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-dc8b9a-dockercfg-bxq2w\"" Jul 29 01:50:18.469596 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.469083 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-dc8b9a\"" Jul 29 01:50:18.473284 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.473259 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz"] Jul 29 01:50:18.477526 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.477506 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x"] Jul 29 01:50:18.481701 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.481679 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x" Jul 29 01:50:18.492027 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.492000 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x"] Jul 29 01:50:18.556094 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.556054 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d512fff-d4f8-4ac0-af92-1023a2dfae3e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x\" (UID: \"5d512fff-d4f8-4ac0-af92-1023a2dfae3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x" Jul 29 01:50:18.556274 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.556142 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz\" (UID: \"691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz" Jul 29 01:50:18.556274 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.556163 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb4cg\" (UniqueName: \"kubernetes.io/projected/5d512fff-d4f8-4ac0-af92-1023a2dfae3e-kube-api-access-pb4cg\") pod \"payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x\" (UID: \"5d512fff-d4f8-4ac0-af92-1023a2dfae3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x" Jul 29 01:50:18.556274 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.556195 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8wxw\" (UniqueName: \"kubernetes.io/projected/691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7-kube-api-access-b8wxw\") pod \"payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz\" (UID: \"691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz" Jul 29 01:50:18.627131 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.627090 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" Jul 29 01:50:18.657014 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.656983 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz\" (UID: \"691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz" Jul 29 01:50:18.657181 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.657019 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb4cg\" (UniqueName: \"kubernetes.io/projected/5d512fff-d4f8-4ac0-af92-1023a2dfae3e-kube-api-access-pb4cg\") pod \"payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x\" (UID: \"5d512fff-d4f8-4ac0-af92-1023a2dfae3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x" Jul 29 01:50:18.657181 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.657053 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8wxw\" (UniqueName: \"kubernetes.io/projected/691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7-kube-api-access-b8wxw\") pod \"payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz\" (UID: \"691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz" Jul 29 01:50:18.657181 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.657107 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d512fff-d4f8-4ac0-af92-1023a2dfae3e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x\" (UID: \"5d512fff-d4f8-4ac0-af92-1023a2dfae3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x" Jul 29 01:50:18.657831 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.657794 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d512fff-d4f8-4ac0-af92-1023a2dfae3e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x\" (UID: \"5d512fff-d4f8-4ac0-af92-1023a2dfae3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x" Jul 29 01:50:18.657944 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.657797 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz\" (UID: \"691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz" Jul 29 01:50:18.665598 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.665570 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8wxw\" (UniqueName: \"kubernetes.io/projected/691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7-kube-api-access-b8wxw\") pod \"payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz\" (UID: \"691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz" Jul 29 01:50:18.665763 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.665739 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb4cg\" (UniqueName: \"kubernetes.io/projected/5d512fff-d4f8-4ac0-af92-1023a2dfae3e-kube-api-access-pb4cg\") pod \"payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x\" (UID: \"5d512fff-d4f8-4ac0-af92-1023a2dfae3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x" Jul 29 01:50:18.766074 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.766045 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4"] Jul 29 01:50:18.768518 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:50:18.768489 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6a9869_1068_4c80_8773_70ebf4f6d1e9.slice/crio-1d53f79288420391f3bec7300e3cecc28314463e10798685510582cd13e39bf9 WatchSource:0}: Error finding container 1d53f79288420391f3bec7300e3cecc28314463e10798685510582cd13e39bf9: Status 404 returned error can't find the container with id 1d53f79288420391f3bec7300e3cecc28314463e10798685510582cd13e39bf9 Jul 29 01:50:18.781242 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.781206 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz" Jul 29 01:50:18.792693 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.792668 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x" Jul 29 01:50:18.927060 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.927029 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz"] Jul 29 01:50:18.928701 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:50:18.928629 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691ae1cc_ab49_4ebb_85ed_3cc73b7e83a7.slice/crio-57d10c4d5c1a5262991cbeae95cef7e9e293e4b022e3d38cfc4d1e6fd048d528 WatchSource:0}: Error finding container 57d10c4d5c1a5262991cbeae95cef7e9e293e4b022e3d38cfc4d1e6fd048d528: Status 404 returned error can't find the container with id 57d10c4d5c1a5262991cbeae95cef7e9e293e4b022e3d38cfc4d1e6fd048d528 Jul 29 01:50:18.953335 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.953313 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x"] Jul 29 01:50:18.969140 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:50:18.969113 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d512fff_d4f8_4ac0_af92_1023a2dfae3e.slice/crio-d5d7be5eb33c2ef3f820bb137142caf5f28d82b4406c426829c703231cdc8c22 WatchSource:0}: Error finding container d5d7be5eb33c2ef3f820bb137142caf5f28d82b4406c426829c703231cdc8c22: Status 404 returned error can't find the container with id d5d7be5eb33c2ef3f820bb137142caf5f28d82b4406c426829c703231cdc8c22 Jul 29 01:50:18.977077 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.977045 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x" event={"ID":"5d512fff-d4f8-4ac0-af92-1023a2dfae3e","Type":"ContainerStarted","Data":"d5d7be5eb33c2ef3f820bb137142caf5f28d82b4406c426829c703231cdc8c22"} Jul 29 01:50:18.978191 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.978164 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" event={"ID":"5a6a9869-1068-4c80-8773-70ebf4f6d1e9","Type":"ContainerStarted","Data":"1d53f79288420391f3bec7300e3cecc28314463e10798685510582cd13e39bf9"} Jul 29 01:50:18.979472 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:18.979441 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz" event={"ID":"691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7","Type":"ContainerStarted","Data":"57d10c4d5c1a5262991cbeae95cef7e9e293e4b022e3d38cfc4d1e6fd048d528"} Jul 29 01:50:19.987257 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:19.987215 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x" event={"ID":"5d512fff-d4f8-4ac0-af92-1023a2dfae3e","Type":"ContainerStarted","Data":"c370515633af12ff653aa309c9318f7b40672edd714b5ac8e6302e3a207df325"} Jul 29 01:50:19.987735 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:19.987310 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x" Jul 29 01:50:19.988733 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:19.988707 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" event={"ID":"5a6a9869-1068-4c80-8773-70ebf4f6d1e9","Type":"ContainerStarted","Data":"a704b078f818d5c5798c48a0c229b16ae4352ef7cb14c092af19c1563155e859"} Jul 29 01:50:19.988863 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:19.988774 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" Jul 29 01:50:19.990086 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:19.990064 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz" event={"ID":"691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7","Type":"ContainerStarted","Data":"542255b2b49e518a4cdc7e5c3255036ae404a95bcb636e87e2904c373b0cc07f"} Jul 29 01:50:19.990240 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:19.990102 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz" Jul 29 01:50:20.003553 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:20.003512 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x" podStartSLOduration=2.003498038 podStartE2EDuration="2.003498038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:50:20.002169374 +0000 UTC m=+3370.272813681" watchObservedRunningTime="2026-07-29 01:50:20.003498038 +0000 UTC m=+3370.274142382" Jul 29 01:50:20.019188 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:20.019137 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" podStartSLOduration=2.563976381 podStartE2EDuration="3.019123903s" totalImagesPullingTime="455.147522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:50:20.017791076 +0000 UTC m=+3370.288435381" watchObservedRunningTime="2026-07-29 01:50:20.019123903 +0000 UTC m=+3370.289768187" Jul 29 01:50:20.035512 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:20.035462 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz" podStartSLOduration=2.03544714 podStartE2EDuration="2.03544714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 01:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:50:20.033251474 +0000 UTC m=+3370.303895783" watchObservedRunningTime="2026-07-29 01:50:20.03544714 +0000 UTC m=+3370.306091448" Jul 29 01:50:26.001298 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:26.001265 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" Jul 29 01:50:30.997387 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:30.997356 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz" Jul 29 01:50:30.997904 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:30.997436 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x" Jul 29 01:50:48.218320 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.218283 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4"] Jul 29 01:50:48.218799 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.218542 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" podUID="5a6a9869-1068-4c80-8773-70ebf4f6d1e9" containerName="maas-api" containerID="cri-o://a704b078f818d5c5798c48a0c229b16ae4352ef7cb14c092af19c1563155e859" gracePeriod=30 Jul 29 01:50:48.278640 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.278603 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x"] Jul 29 01:50:48.278947 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.278895 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x" podUID="5d512fff-d4f8-4ac0-af92-1023a2dfae3e" containerName="payload-processing" containerID="cri-o://c370515633af12ff653aa309c9318f7b40672edd714b5ac8e6302e3a207df325" gracePeriod=30 Jul 29 01:50:48.286538 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.286486 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz"] Jul 29 01:50:48.287143 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.286751 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz" podUID="691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7" containerName="payload-pre-processing" containerID="cri-o://542255b2b49e518a4cdc7e5c3255036ae404a95bcb636e87e2904c373b0cc07f" gracePeriod=30 Jul 29 01:50:48.568829 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.568791 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x" Jul 29 01:50:48.596241 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.596216 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" Jul 29 01:50:48.608657 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.608635 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz" Jul 29 01:50:48.650294 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.650258 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a6a9869-1068-4c80-8773-70ebf4f6d1e9-maas-api-tls\") pod \"5a6a9869-1068-4c80-8773-70ebf4f6d1e9\" (UID: \"5a6a9869-1068-4c80-8773-70ebf4f6d1e9\") " Jul 29 01:50:48.650294 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.650289 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8wxw\" (UniqueName: \"kubernetes.io/projected/691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7-kube-api-access-b8wxw\") pod \"691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7\" (UID: \"691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7\") " Jul 29 01:50:48.650541 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.650324 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7-plugins-config-volume\") pod \"691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7\" (UID: \"691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7\") " Jul 29 01:50:48.650541 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.650347 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d512fff-d4f8-4ac0-af92-1023a2dfae3e-plugins-config-volume\") pod \"5d512fff-d4f8-4ac0-af92-1023a2dfae3e\" (UID: \"5d512fff-d4f8-4ac0-af92-1023a2dfae3e\") " Jul 29 01:50:48.650541 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.650389 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb4cg\" (UniqueName: \"kubernetes.io/projected/5d512fff-d4f8-4ac0-af92-1023a2dfae3e-kube-api-access-pb4cg\") pod \"5d512fff-d4f8-4ac0-af92-1023a2dfae3e\" (UID: \"5d512fff-d4f8-4ac0-af92-1023a2dfae3e\") " Jul 29 01:50:48.650541 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.650480 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a6a9869-1068-4c80-8773-70ebf4f6d1e9-openshift-service-ca\") pod \"5a6a9869-1068-4c80-8773-70ebf4f6d1e9\" (UID: \"5a6a9869-1068-4c80-8773-70ebf4f6d1e9\") " Jul 29 01:50:48.650541 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.650509 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhngv\" (UniqueName: \"kubernetes.io/projected/5a6a9869-1068-4c80-8773-70ebf4f6d1e9-kube-api-access-dhngv\") pod \"5a6a9869-1068-4c80-8773-70ebf4f6d1e9\" (UID: \"5a6a9869-1068-4c80-8773-70ebf4f6d1e9\") " Jul 29 01:50:48.650853 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.650698 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7" (UID: "691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:50:48.650853 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.650794 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d512fff-d4f8-4ac0-af92-1023a2dfae3e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d512fff-d4f8-4ac0-af92-1023a2dfae3e" (UID: "5d512fff-d4f8-4ac0-af92-1023a2dfae3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:50:48.651019 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.650993 2543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7-plugins-config-volume\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:50:48.651177 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.651132 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a6a9869-1068-4c80-8773-70ebf4f6d1e9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5a6a9869-1068-4c80-8773-70ebf4f6d1e9" (UID: "5a6a9869-1068-4c80-8773-70ebf4f6d1e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:50:48.652638 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.652608 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a6a9869-1068-4c80-8773-70ebf4f6d1e9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5a6a9869-1068-4c80-8773-70ebf4f6d1e9" (UID: "5a6a9869-1068-4c80-8773-70ebf4f6d1e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 01:50:48.652748 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.652699 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7-kube-api-access-b8wxw" (OuterVolumeSpecName: "kube-api-access-b8wxw") pod "691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7" (UID: "691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7"). InnerVolumeSpecName "kube-api-access-b8wxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:50:48.652748 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.652717 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6a9869-1068-4c80-8773-70ebf4f6d1e9-kube-api-access-dhngv" (OuterVolumeSpecName: "kube-api-access-dhngv") pod "5a6a9869-1068-4c80-8773-70ebf4f6d1e9" (UID: "5a6a9869-1068-4c80-8773-70ebf4f6d1e9"). InnerVolumeSpecName "kube-api-access-dhngv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:50:48.653173 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.653149 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d512fff-d4f8-4ac0-af92-1023a2dfae3e-kube-api-access-pb4cg" (OuterVolumeSpecName: "kube-api-access-pb4cg") pod "5d512fff-d4f8-4ac0-af92-1023a2dfae3e" (UID: "5d512fff-d4f8-4ac0-af92-1023a2dfae3e"). InnerVolumeSpecName "kube-api-access-pb4cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:50:48.751634 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.751600 2543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a6a9869-1068-4c80-8773-70ebf4f6d1e9-openshift-service-ca\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:50:48.751634 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.751630 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhngv\" (UniqueName: \"kubernetes.io/projected/5a6a9869-1068-4c80-8773-70ebf4f6d1e9-kube-api-access-dhngv\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:50:48.751634 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.751641 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a6a9869-1068-4c80-8773-70ebf4f6d1e9-maas-api-tls\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:50:48.751905 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.751651 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8wxw\" (UniqueName: \"kubernetes.io/projected/691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7-kube-api-access-b8wxw\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:50:48.751905 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.751661 2543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d512fff-d4f8-4ac0-af92-1023a2dfae3e-plugins-config-volume\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:50:48.751905 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:48.751669 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb4cg\" (UniqueName: \"kubernetes.io/projected/5d512fff-d4f8-4ac0-af92-1023a2dfae3e-kube-api-access-pb4cg\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:50:49.150416 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.150380 2543 generic.go:356] "Generic (PLEG): container finished" podID="691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7" containerID="542255b2b49e518a4cdc7e5c3255036ae404a95bcb636e87e2904c373b0cc07f" exitCode=0 Jul 29 01:50:49.150602 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.150443 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz" Jul 29 01:50:49.150602 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.150458 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz" event={"ID":"691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7","Type":"ContainerDied","Data":"542255b2b49e518a4cdc7e5c3255036ae404a95bcb636e87e2904c373b0cc07f"} Jul 29 01:50:49.150602 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.150550 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz" event={"ID":"691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7","Type":"ContainerDied","Data":"57d10c4d5c1a5262991cbeae95cef7e9e293e4b022e3d38cfc4d1e6fd048d528"} Jul 29 01:50:49.150602 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.150578 2543 scope.go:117] "RemoveContainer" containerID="542255b2b49e518a4cdc7e5c3255036ae404a95bcb636e87e2904c373b0cc07f" Jul 29 01:50:49.151872 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.151788 2543 generic.go:356] "Generic (PLEG): container finished" podID="5d512fff-d4f8-4ac0-af92-1023a2dfae3e" containerID="c370515633af12ff653aa309c9318f7b40672edd714b5ac8e6302e3a207df325" exitCode=0 Jul 29 01:50:49.151872 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.151855 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x" event={"ID":"5d512fff-d4f8-4ac0-af92-1023a2dfae3e","Type":"ContainerDied","Data":"c370515633af12ff653aa309c9318f7b40672edd714b5ac8e6302e3a207df325"} Jul 29 01:50:49.151872 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.151862 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x" Jul 29 01:50:49.152098 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.151925 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x" event={"ID":"5d512fff-d4f8-4ac0-af92-1023a2dfae3e","Type":"ContainerDied","Data":"d5d7be5eb33c2ef3f820bb137142caf5f28d82b4406c426829c703231cdc8c22"} Jul 29 01:50:49.153492 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.153433 2543 generic.go:356] "Generic (PLEG): container finished" podID="5a6a9869-1068-4c80-8773-70ebf4f6d1e9" containerID="a704b078f818d5c5798c48a0c229b16ae4352ef7cb14c092af19c1563155e859" exitCode=0 Jul 29 01:50:49.153492 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.153461 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" event={"ID":"5a6a9869-1068-4c80-8773-70ebf4f6d1e9","Type":"ContainerDied","Data":"a704b078f818d5c5798c48a0c229b16ae4352ef7cb14c092af19c1563155e859"} Jul 29 01:50:49.153680 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.153495 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" Jul 29 01:50:49.153680 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.153507 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4" event={"ID":"5a6a9869-1068-4c80-8773-70ebf4f6d1e9","Type":"ContainerDied","Data":"1d53f79288420391f3bec7300e3cecc28314463e10798685510582cd13e39bf9"} Jul 29 01:50:49.160398 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.160388 2543 scope.go:117] "RemoveContainer" containerID="542255b2b49e518a4cdc7e5c3255036ae404a95bcb636e87e2904c373b0cc07f" Jul 29 01:50:49.160651 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:50:49.160630 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"542255b2b49e518a4cdc7e5c3255036ae404a95bcb636e87e2904c373b0cc07f\": container with ID starting with 542255b2b49e518a4cdc7e5c3255036ae404a95bcb636e87e2904c373b0cc07f not found: ID does not exist" containerID="542255b2b49e518a4cdc7e5c3255036ae404a95bcb636e87e2904c373b0cc07f" Jul 29 01:50:49.160751 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.160660 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"542255b2b49e518a4cdc7e5c3255036ae404a95bcb636e87e2904c373b0cc07f"} err="failed to get container status \"542255b2b49e518a4cdc7e5c3255036ae404a95bcb636e87e2904c373b0cc07f\": rpc error: code = NotFound desc = could not find container \"542255b2b49e518a4cdc7e5c3255036ae404a95bcb636e87e2904c373b0cc07f\": container with ID starting with 542255b2b49e518a4cdc7e5c3255036ae404a95bcb636e87e2904c373b0cc07f not found: ID does not exist" Jul 29 01:50:49.160751 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.160681 2543 scope.go:117] "RemoveContainer" containerID="c370515633af12ff653aa309c9318f7b40672edd714b5ac8e6302e3a207df325" Jul 29 01:50:49.169732 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.169716 2543 scope.go:117] "RemoveContainer" containerID="c370515633af12ff653aa309c9318f7b40672edd714b5ac8e6302e3a207df325" Jul 29 01:50:49.169993 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:50:49.169975 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c370515633af12ff653aa309c9318f7b40672edd714b5ac8e6302e3a207df325\": container with ID starting with c370515633af12ff653aa309c9318f7b40672edd714b5ac8e6302e3a207df325 not found: ID does not exist" containerID="c370515633af12ff653aa309c9318f7b40672edd714b5ac8e6302e3a207df325" Jul 29 01:50:49.170086 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.169999 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c370515633af12ff653aa309c9318f7b40672edd714b5ac8e6302e3a207df325"} err="failed to get container status \"c370515633af12ff653aa309c9318f7b40672edd714b5ac8e6302e3a207df325\": rpc error: code = NotFound desc = could not find container \"c370515633af12ff653aa309c9318f7b40672edd714b5ac8e6302e3a207df325\": container with ID starting with c370515633af12ff653aa309c9318f7b40672edd714b5ac8e6302e3a207df325 not found: ID does not exist" Jul 29 01:50:49.170086 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.170015 2543 scope.go:117] "RemoveContainer" containerID="a704b078f818d5c5798c48a0c229b16ae4352ef7cb14c092af19c1563155e859" Jul 29 01:50:49.173107 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.173088 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz"] Jul 29 01:50:49.177383 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.177362 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dc8b9a-96fcd9f84-5tvzz"] Jul 29 01:50:49.178724 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.178704 2543 scope.go:117] "RemoveContainer" containerID="a704b078f818d5c5798c48a0c229b16ae4352ef7cb14c092af19c1563155e859" Jul 29 01:50:49.179001 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:50:49.178982 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a704b078f818d5c5798c48a0c229b16ae4352ef7cb14c092af19c1563155e859\": container with ID starting with a704b078f818d5c5798c48a0c229b16ae4352ef7cb14c092af19c1563155e859 not found: ID does not exist" containerID="a704b078f818d5c5798c48a0c229b16ae4352ef7cb14c092af19c1563155e859" Jul 29 01:50:49.179067 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.179007 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a704b078f818d5c5798c48a0c229b16ae4352ef7cb14c092af19c1563155e859"} err="failed to get container status \"a704b078f818d5c5798c48a0c229b16ae4352ef7cb14c092af19c1563155e859\": rpc error: code = NotFound desc = could not find container \"a704b078f818d5c5798c48a0c229b16ae4352ef7cb14c092af19c1563155e859\": container with ID starting with a704b078f818d5c5798c48a0c229b16ae4352ef7cb14c092af19c1563155e859 not found: ID does not exist" Jul 29 01:50:49.188536 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.188515 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4"] Jul 29 01:50:49.193576 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.193557 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dc8b9a-7cc984b699-77kw4"] Jul 29 01:50:49.207290 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.207267 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x"] Jul 29 01:50:49.213401 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:49.213379 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-dc8b9a-7866f5ff78-xg98x"] Jul 29 01:50:50.342922 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:50.342888 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a6a9869-1068-4c80-8773-70ebf4f6d1e9" path="/var/lib/kubelet/pods/5a6a9869-1068-4c80-8773-70ebf4f6d1e9/volumes" Jul 29 01:50:50.343466 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:50.343448 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d512fff-d4f8-4ac0-af92-1023a2dfae3e" path="/var/lib/kubelet/pods/5d512fff-d4f8-4ac0-af92-1023a2dfae3e/volumes" Jul 29 01:50:50.343847 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:50:50.343833 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7" path="/var/lib/kubelet/pods/691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7/volumes" Jul 29 01:51:14.016992 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.016956 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq"] Jul 29 01:51:14.017525 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.017166 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq" podUID="cb63cf49-dd44-44e4-8564-3be4e2360251" containerName="maas-api" containerID="cri-o://8be92792afb147805c5996d63b693b1dba27c9c9d614786332ef5d1bfb0e7791" gracePeriod=30 Jul 29 01:51:14.098038 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.097976 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957"] Jul 29 01:51:14.106570 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.106538 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw"] Jul 29 01:51:14.106838 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.106788 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw" podUID="4a31040a-c8a7-45ac-b508-5cac456360a6" containerName="payload-pre-processing" containerID="cri-o://924740bda7dab33258bf1601a57d66051451f07efbc68de118939b23d0579d4f" gracePeriod=30 Jul 29 01:51:14.315582 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.307188 2543 generic.go:356] "Generic (PLEG): container finished" podID="cb63cf49-dd44-44e4-8564-3be4e2360251" containerID="8be92792afb147805c5996d63b693b1dba27c9c9d614786332ef5d1bfb0e7791" exitCode=0 Jul 29 01:51:14.315582 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.307338 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq" event={"ID":"cb63cf49-dd44-44e4-8564-3be4e2360251","Type":"ContainerDied","Data":"8be92792afb147805c5996d63b693b1dba27c9c9d614786332ef5d1bfb0e7791"} Jul 29 01:51:14.322649 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.322577 2543 generic.go:356] "Generic (PLEG): container finished" podID="4a31040a-c8a7-45ac-b508-5cac456360a6" containerID="924740bda7dab33258bf1601a57d66051451f07efbc68de118939b23d0579d4f" exitCode=0 Jul 29 01:51:14.322891 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.322656 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw" event={"ID":"4a31040a-c8a7-45ac-b508-5cac456360a6","Type":"ContainerDied","Data":"924740bda7dab33258bf1601a57d66051451f07efbc68de118939b23d0579d4f"} Jul 29 01:51:14.322891 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.322802 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957" podUID="3f6cf9cc-204c-4ac0-8889-81eb506a1375" containerName="payload-processing" containerID="cri-o://b30d3f9f8e7bbced5ca8358a0ee800d33de0efdca693afdae9052f7a6b658d7a" gracePeriod=30 Jul 29 01:51:14.435950 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.435926 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw" Jul 29 01:51:14.438543 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.438519 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq" Jul 29 01:51:14.589467 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.589445 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957" Jul 29 01:51:14.602859 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.602833 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b25p2\" (UniqueName: \"kubernetes.io/projected/4a31040a-c8a7-45ac-b508-5cac456360a6-kube-api-access-b25p2\") pod \"4a31040a-c8a7-45ac-b508-5cac456360a6\" (UID: \"4a31040a-c8a7-45ac-b508-5cac456360a6\") " Jul 29 01:51:14.602996 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.602893 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb63cf49-dd44-44e4-8564-3be4e2360251-openshift-service-ca\") pod \"cb63cf49-dd44-44e4-8564-3be4e2360251\" (UID: \"cb63cf49-dd44-44e4-8564-3be4e2360251\") " Jul 29 01:51:14.602996 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.602917 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bttm2\" (UniqueName: \"kubernetes.io/projected/cb63cf49-dd44-44e4-8564-3be4e2360251-kube-api-access-bttm2\") pod \"cb63cf49-dd44-44e4-8564-3be4e2360251\" (UID: \"cb63cf49-dd44-44e4-8564-3be4e2360251\") " Jul 29 01:51:14.602996 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.602954 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb63cf49-dd44-44e4-8564-3be4e2360251-maas-api-tls\") pod \"cb63cf49-dd44-44e4-8564-3be4e2360251\" (UID: \"cb63cf49-dd44-44e4-8564-3be4e2360251\") " Jul 29 01:51:14.602996 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.602981 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a31040a-c8a7-45ac-b508-5cac456360a6-plugins-config-volume\") pod \"4a31040a-c8a7-45ac-b508-5cac456360a6\" (UID: \"4a31040a-c8a7-45ac-b508-5cac456360a6\") " Jul 29 01:51:14.603342 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.603315 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb63cf49-dd44-44e4-8564-3be4e2360251-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cb63cf49-dd44-44e4-8564-3be4e2360251" (UID: "cb63cf49-dd44-44e4-8564-3be4e2360251"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:51:14.603446 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.603357 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a31040a-c8a7-45ac-b508-5cac456360a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4a31040a-c8a7-45ac-b508-5cac456360a6" (UID: "4a31040a-c8a7-45ac-b508-5cac456360a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:51:14.605767 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.605156 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb63cf49-dd44-44e4-8564-3be4e2360251-kube-api-access-bttm2" (OuterVolumeSpecName: "kube-api-access-bttm2") pod "cb63cf49-dd44-44e4-8564-3be4e2360251" (UID: "cb63cf49-dd44-44e4-8564-3be4e2360251"). InnerVolumeSpecName "kube-api-access-bttm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:51:14.605767 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.605439 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb63cf49-dd44-44e4-8564-3be4e2360251-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cb63cf49-dd44-44e4-8564-3be4e2360251" (UID: "cb63cf49-dd44-44e4-8564-3be4e2360251"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 01:51:14.605930 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.605854 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a31040a-c8a7-45ac-b508-5cac456360a6-kube-api-access-b25p2" (OuterVolumeSpecName: "kube-api-access-b25p2") pod "4a31040a-c8a7-45ac-b508-5cac456360a6" (UID: "4a31040a-c8a7-45ac-b508-5cac456360a6"). InnerVolumeSpecName "kube-api-access-b25p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:51:14.704539 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.704499 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f6cf9cc-204c-4ac0-8889-81eb506a1375-plugins-config-volume\") pod \"3f6cf9cc-204c-4ac0-8889-81eb506a1375\" (UID: \"3f6cf9cc-204c-4ac0-8889-81eb506a1375\") " Jul 29 01:51:14.704744 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.704581 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rkkj\" (UniqueName: \"kubernetes.io/projected/3f6cf9cc-204c-4ac0-8889-81eb506a1375-kube-api-access-2rkkj\") pod \"3f6cf9cc-204c-4ac0-8889-81eb506a1375\" (UID: \"3f6cf9cc-204c-4ac0-8889-81eb506a1375\") " Jul 29 01:51:14.704844 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.704781 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb63cf49-dd44-44e4-8564-3be4e2360251-maas-api-tls\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:51:14.704844 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.704792 2543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a31040a-c8a7-45ac-b508-5cac456360a6-plugins-config-volume\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:51:14.704844 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.704801 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b25p2\" (UniqueName: \"kubernetes.io/projected/4a31040a-c8a7-45ac-b508-5cac456360a6-kube-api-access-b25p2\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:51:14.704844 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.704810 2543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb63cf49-dd44-44e4-8564-3be4e2360251-openshift-service-ca\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:51:14.705032 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.704851 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bttm2\" (UniqueName: \"kubernetes.io/projected/cb63cf49-dd44-44e4-8564-3be4e2360251-kube-api-access-bttm2\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:51:14.705032 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.704873 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f6cf9cc-204c-4ac0-8889-81eb506a1375-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3f6cf9cc-204c-4ac0-8889-81eb506a1375" (UID: "3f6cf9cc-204c-4ac0-8889-81eb506a1375"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:51:14.706628 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.706604 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f6cf9cc-204c-4ac0-8889-81eb506a1375-kube-api-access-2rkkj" (OuterVolumeSpecName: "kube-api-access-2rkkj") pod "3f6cf9cc-204c-4ac0-8889-81eb506a1375" (UID: "3f6cf9cc-204c-4ac0-8889-81eb506a1375"). InnerVolumeSpecName "kube-api-access-2rkkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:51:14.805367 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.805320 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rkkj\" (UniqueName: \"kubernetes.io/projected/3f6cf9cc-204c-4ac0-8889-81eb506a1375-kube-api-access-2rkkj\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:51:14.805367 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:14.805360 2543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f6cf9cc-204c-4ac0-8889-81eb506a1375-plugins-config-volume\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:51:15.328521 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:15.328490 2543 generic.go:356] "Generic (PLEG): container finished" podID="3f6cf9cc-204c-4ac0-8889-81eb506a1375" containerID="b30d3f9f8e7bbced5ca8358a0ee800d33de0efdca693afdae9052f7a6b658d7a" exitCode=0 Jul 29 01:51:15.328961 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:15.328538 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957" Jul 29 01:51:15.328961 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:15.328573 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957" event={"ID":"3f6cf9cc-204c-4ac0-8889-81eb506a1375","Type":"ContainerDied","Data":"b30d3f9f8e7bbced5ca8358a0ee800d33de0efdca693afdae9052f7a6b658d7a"} Jul 29 01:51:15.328961 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:15.328625 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957" event={"ID":"3f6cf9cc-204c-4ac0-8889-81eb506a1375","Type":"ContainerDied","Data":"152b7ff0ed70b5d988a911b3515fd2d9cb5c79d0250dd77bf1aca442628f0a1a"} Jul 29 01:51:15.328961 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:15.328658 2543 scope.go:117] "RemoveContainer" containerID="b30d3f9f8e7bbced5ca8358a0ee800d33de0efdca693afdae9052f7a6b658d7a" Jul 29 01:51:15.329977 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:15.329937 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq" Jul 29 01:51:15.330054 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:15.329969 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq" event={"ID":"cb63cf49-dd44-44e4-8564-3be4e2360251","Type":"ContainerDied","Data":"bcefb6748477b17e41297ae3b795f23f23a8dccd73b1760b3f8acf69aed5bbd8"} Jul 29 01:51:15.331432 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:15.331410 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw" Jul 29 01:51:15.331432 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:15.331422 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw" event={"ID":"4a31040a-c8a7-45ac-b508-5cac456360a6","Type":"ContainerDied","Data":"607caeddce3dfd3d48163ea9c78c548c756e45ca73ed7ec01e0489194be16441"} Jul 29 01:51:15.341483 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:15.341032 2543 scope.go:117] "RemoveContainer" containerID="b30d3f9f8e7bbced5ca8358a0ee800d33de0efdca693afdae9052f7a6b658d7a" Jul 29 01:51:15.341483 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:51:15.341372 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b30d3f9f8e7bbced5ca8358a0ee800d33de0efdca693afdae9052f7a6b658d7a\": container with ID starting with b30d3f9f8e7bbced5ca8358a0ee800d33de0efdca693afdae9052f7a6b658d7a not found: ID does not exist" containerID="b30d3f9f8e7bbced5ca8358a0ee800d33de0efdca693afdae9052f7a6b658d7a" Jul 29 01:51:15.341483 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:15.341409 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30d3f9f8e7bbced5ca8358a0ee800d33de0efdca693afdae9052f7a6b658d7a"} err="failed to get container status \"b30d3f9f8e7bbced5ca8358a0ee800d33de0efdca693afdae9052f7a6b658d7a\": rpc error: code = NotFound desc = could not find container \"b30d3f9f8e7bbced5ca8358a0ee800d33de0efdca693afdae9052f7a6b658d7a\": container with ID starting with b30d3f9f8e7bbced5ca8358a0ee800d33de0efdca693afdae9052f7a6b658d7a not found: ID does not exist" Jul 29 01:51:15.341483 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:15.341431 2543 scope.go:117] "RemoveContainer" containerID="8be92792afb147805c5996d63b693b1dba27c9c9d614786332ef5d1bfb0e7791" Jul 29 01:51:15.353247 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:15.353231 2543 scope.go:117] "RemoveContainer" containerID="924740bda7dab33258bf1601a57d66051451f07efbc68de118939b23d0579d4f" Jul 29 01:51:15.361416 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:15.361387 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957"] Jul 29 01:51:15.364576 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:15.364556 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-295648-6b6b77645d-zh957"] Jul 29 01:51:15.375769 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:15.375747 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq"] Jul 29 01:51:15.383132 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:15.383100 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-295648-5f7fd75497-v2xlq"] Jul 29 01:51:15.394384 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:15.394363 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw"] Jul 29 01:51:15.398554 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:15.398532 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-295648-78684596c6-gbcjw"] Jul 29 01:51:16.343150 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:16.343118 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f6cf9cc-204c-4ac0-8889-81eb506a1375" path="/var/lib/kubelet/pods/3f6cf9cc-204c-4ac0-8889-81eb506a1375/volumes" Jul 29 01:51:16.343594 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:16.343580 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a31040a-c8a7-45ac-b508-5cac456360a6" path="/var/lib/kubelet/pods/4a31040a-c8a7-45ac-b508-5cac456360a6/volumes" Jul 29 01:51:16.343963 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:51:16.343949 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb63cf49-dd44-44e4-8564-3be4e2360251" path="/var/lib/kubelet/pods/cb63cf49-dd44-44e4-8564-3be4e2360251/volumes" Jul 29 01:52:14.152402 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.152363 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k"] Jul 29 01:52:14.152988 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.152665 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" podUID="c283ffd5-04ec-4c64-8019-45da4b5aad0b" containerName="maas-api" containerID="cri-o://0c52c331bc6d402d62005743c4dc2a04270f0b8f2677fc38252c2445434299a9" gracePeriod=30 Jul 29 01:52:14.214189 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.213899 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph"] Jul 29 01:52:14.214189 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.214157 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph" podUID="9b0743f8-60ec-4f0f-a2a5-98ab5f206db7" containerName="payload-processing" containerID="cri-o://6ca09207ffab2b200540d39e69dc808f89dead991a5f6c30093354e7d8e6020e" gracePeriod=30 Jul 29 01:52:14.495678 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.495651 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" Jul 29 01:52:14.498564 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.498536 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph" Jul 29 01:52:14.561337 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.561307 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6b8tz\" (UniqueName: \"kubernetes.io/projected/9b0743f8-60ec-4f0f-a2a5-98ab5f206db7-kube-api-access-6b8tz\") pod \"9b0743f8-60ec-4f0f-a2a5-98ab5f206db7\" (UID: \"9b0743f8-60ec-4f0f-a2a5-98ab5f206db7\") " Jul 29 01:52:14.561537 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.561365 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0743f8-60ec-4f0f-a2a5-98ab5f206db7-plugins-config-volume\") pod \"9b0743f8-60ec-4f0f-a2a5-98ab5f206db7\" (UID: \"9b0743f8-60ec-4f0f-a2a5-98ab5f206db7\") " Jul 29 01:52:14.561537 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.561399 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c283ffd5-04ec-4c64-8019-45da4b5aad0b-openshift-service-ca\") pod \"c283ffd5-04ec-4c64-8019-45da4b5aad0b\" (UID: \"c283ffd5-04ec-4c64-8019-45da4b5aad0b\") " Jul 29 01:52:14.561537 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.561466 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snxfm\" (UniqueName: \"kubernetes.io/projected/c283ffd5-04ec-4c64-8019-45da4b5aad0b-kube-api-access-snxfm\") pod \"c283ffd5-04ec-4c64-8019-45da4b5aad0b\" (UID: \"c283ffd5-04ec-4c64-8019-45da4b5aad0b\") " Jul 29 01:52:14.561537 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.561495 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c283ffd5-04ec-4c64-8019-45da4b5aad0b-maas-api-tls\") pod \"c283ffd5-04ec-4c64-8019-45da4b5aad0b\" (UID: \"c283ffd5-04ec-4c64-8019-45da4b5aad0b\") " Jul 29 01:52:14.561789 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.561762 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0743f8-60ec-4f0f-a2a5-98ab5f206db7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9b0743f8-60ec-4f0f-a2a5-98ab5f206db7" (UID: "9b0743f8-60ec-4f0f-a2a5-98ab5f206db7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:52:14.561866 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.561794 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c283ffd5-04ec-4c64-8019-45da4b5aad0b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c283ffd5-04ec-4c64-8019-45da4b5aad0b" (UID: "c283ffd5-04ec-4c64-8019-45da4b5aad0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:52:14.563562 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.563540 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0743f8-60ec-4f0f-a2a5-98ab5f206db7-kube-api-access-6b8tz" (OuterVolumeSpecName: "kube-api-access-6b8tz") pod "9b0743f8-60ec-4f0f-a2a5-98ab5f206db7" (UID: "9b0743f8-60ec-4f0f-a2a5-98ab5f206db7"). InnerVolumeSpecName "kube-api-access-6b8tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:52:14.563660 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.563605 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c283ffd5-04ec-4c64-8019-45da4b5aad0b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c283ffd5-04ec-4c64-8019-45da4b5aad0b" (UID: "c283ffd5-04ec-4c64-8019-45da4b5aad0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 01:52:14.563705 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.563678 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c283ffd5-04ec-4c64-8019-45da4b5aad0b-kube-api-access-snxfm" (OuterVolumeSpecName: "kube-api-access-snxfm") pod "c283ffd5-04ec-4c64-8019-45da4b5aad0b" (UID: "c283ffd5-04ec-4c64-8019-45da4b5aad0b"). InnerVolumeSpecName "kube-api-access-snxfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:52:14.644779 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.644746 2543 generic.go:356] "Generic (PLEG): container finished" podID="c283ffd5-04ec-4c64-8019-45da4b5aad0b" containerID="0c52c331bc6d402d62005743c4dc2a04270f0b8f2677fc38252c2445434299a9" exitCode=0 Jul 29 01:52:14.644993 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.644789 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" Jul 29 01:52:14.644993 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.644855 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" event={"ID":"c283ffd5-04ec-4c64-8019-45da4b5aad0b","Type":"ContainerDied","Data":"0c52c331bc6d402d62005743c4dc2a04270f0b8f2677fc38252c2445434299a9"} Jul 29 01:52:14.644993 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.644912 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k" event={"ID":"c283ffd5-04ec-4c64-8019-45da4b5aad0b","Type":"ContainerDied","Data":"2bce3a9ba55014076d68ac1e1e8dd13feaec8a4c2ccb6e529adaf1e1fbd6ac27"} Jul 29 01:52:14.644993 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.644935 2543 scope.go:117] "RemoveContainer" containerID="0c52c331bc6d402d62005743c4dc2a04270f0b8f2677fc38252c2445434299a9" Jul 29 01:52:14.646163 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.646132 2543 generic.go:356] "Generic (PLEG): container finished" podID="9b0743f8-60ec-4f0f-a2a5-98ab5f206db7" containerID="6ca09207ffab2b200540d39e69dc808f89dead991a5f6c30093354e7d8e6020e" exitCode=0 Jul 29 01:52:14.646245 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.646173 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph" Jul 29 01:52:14.646245 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.646173 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph" event={"ID":"9b0743f8-60ec-4f0f-a2a5-98ab5f206db7","Type":"ContainerDied","Data":"6ca09207ffab2b200540d39e69dc808f89dead991a5f6c30093354e7d8e6020e"} Jul 29 01:52:14.646331 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.646273 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph" event={"ID":"9b0743f8-60ec-4f0f-a2a5-98ab5f206db7","Type":"ContainerDied","Data":"732bfcc331ebfce1b69d01c64908d527ef850de4899c44188505d987cbe63655"} Jul 29 01:52:14.655081 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.655061 2543 scope.go:117] "RemoveContainer" containerID="0c52c331bc6d402d62005743c4dc2a04270f0b8f2677fc38252c2445434299a9" Jul 29 01:52:14.655414 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:52:14.655391 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c52c331bc6d402d62005743c4dc2a04270f0b8f2677fc38252c2445434299a9\": container with ID starting with 0c52c331bc6d402d62005743c4dc2a04270f0b8f2677fc38252c2445434299a9 not found: ID does not exist" containerID="0c52c331bc6d402d62005743c4dc2a04270f0b8f2677fc38252c2445434299a9" Jul 29 01:52:14.655493 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.655435 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c52c331bc6d402d62005743c4dc2a04270f0b8f2677fc38252c2445434299a9"} err="failed to get container status \"0c52c331bc6d402d62005743c4dc2a04270f0b8f2677fc38252c2445434299a9\": rpc error: code = NotFound desc = could not find container \"0c52c331bc6d402d62005743c4dc2a04270f0b8f2677fc38252c2445434299a9\": container with ID starting with 0c52c331bc6d402d62005743c4dc2a04270f0b8f2677fc38252c2445434299a9 not found: ID does not exist" Jul 29 01:52:14.655493 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.655458 2543 scope.go:117] "RemoveContainer" containerID="6ca09207ffab2b200540d39e69dc808f89dead991a5f6c30093354e7d8e6020e" Jul 29 01:52:14.662203 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.662182 2543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0743f8-60ec-4f0f-a2a5-98ab5f206db7-plugins-config-volume\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:52:14.662203 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.662203 2543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c283ffd5-04ec-4c64-8019-45da4b5aad0b-openshift-service-ca\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:52:14.662333 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.662213 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snxfm\" (UniqueName: \"kubernetes.io/projected/c283ffd5-04ec-4c64-8019-45da4b5aad0b-kube-api-access-snxfm\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:52:14.662333 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.662222 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c283ffd5-04ec-4c64-8019-45da4b5aad0b-maas-api-tls\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:52:14.662333 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.662230 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b8tz\" (UniqueName: \"kubernetes.io/projected/9b0743f8-60ec-4f0f-a2a5-98ab5f206db7-kube-api-access-6b8tz\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:52:14.664238 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.664222 2543 scope.go:117] "RemoveContainer" containerID="6ca09207ffab2b200540d39e69dc808f89dead991a5f6c30093354e7d8e6020e" Jul 29 01:52:14.664505 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:52:14.664490 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca09207ffab2b200540d39e69dc808f89dead991a5f6c30093354e7d8e6020e\": container with ID starting with 6ca09207ffab2b200540d39e69dc808f89dead991a5f6c30093354e7d8e6020e not found: ID does not exist" containerID="6ca09207ffab2b200540d39e69dc808f89dead991a5f6c30093354e7d8e6020e" Jul 29 01:52:14.664549 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.664513 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca09207ffab2b200540d39e69dc808f89dead991a5f6c30093354e7d8e6020e"} err="failed to get container status \"6ca09207ffab2b200540d39e69dc808f89dead991a5f6c30093354e7d8e6020e\": rpc error: code = NotFound desc = could not find container \"6ca09207ffab2b200540d39e69dc808f89dead991a5f6c30093354e7d8e6020e\": container with ID starting with 6ca09207ffab2b200540d39e69dc808f89dead991a5f6c30093354e7d8e6020e not found: ID does not exist" Jul 29 01:52:14.685736 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.685652 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k"] Jul 29 01:52:14.687642 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.687623 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8230fd-7f77c4b89f-9v79k"] Jul 29 01:52:14.707649 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.707609 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph"] Jul 29 01:52:14.717568 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:14.717534 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8230fd-689d74946-g2rph"] Jul 29 01:52:16.343134 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:16.343094 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b0743f8-60ec-4f0f-a2a5-98ab5f206db7" path="/var/lib/kubelet/pods/9b0743f8-60ec-4f0f-a2a5-98ab5f206db7/volumes" Jul 29 01:52:16.343520 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:16.343492 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c283ffd5-04ec-4c64-8019-45da4b5aad0b" path="/var/lib/kubelet/pods/c283ffd5-04ec-4c64-8019-45da4b5aad0b/volumes" Jul 29 01:52:31.750719 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:31.750681 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl"] Jul 29 01:52:31.751277 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:31.750899 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" podUID="24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1" containerName="maas-api" containerID="cri-o://cf1ead341346840c6b153aaa0783370e45bf078614b2e54de3f0b6e5417186d0" gracePeriod=30 Jul 29 01:52:31.768692 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:31.768666 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ea1e0-29754825-dnz9n"] Jul 29 01:52:31.774135 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:31.774112 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5ea1e0-29754825-dnz9n"] Jul 29 01:52:31.811605 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:31.811574 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw"] Jul 29 01:52:31.811845 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:31.811791 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw" podUID="eb62aecb-6dab-41b6-a34f-1998e886ac98" containerName="payload-pre-processing" containerID="cri-o://c0e36400a3e3a7f5cdda54edbe8b14225a83d86f4feb72e8f11ec735a2ee0d33" gracePeriod=30 Jul 29 01:52:32.080008 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.079981 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" Jul 29 01:52:32.116624 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.116601 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw" Jul 29 01:52:32.226702 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.226664 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1-maas-api-tls\") pod \"24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1\" (UID: \"24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1\") " Jul 29 01:52:32.226934 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.226727 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snhch\" (UniqueName: \"kubernetes.io/projected/eb62aecb-6dab-41b6-a34f-1998e886ac98-kube-api-access-snhch\") pod \"eb62aecb-6dab-41b6-a34f-1998e886ac98\" (UID: \"eb62aecb-6dab-41b6-a34f-1998e886ac98\") " Jul 29 01:52:32.226934 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.226755 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgcwt\" (UniqueName: \"kubernetes.io/projected/24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1-kube-api-access-kgcwt\") pod \"24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1\" (UID: \"24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1\") " Jul 29 01:52:32.226934 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.226787 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb62aecb-6dab-41b6-a34f-1998e886ac98-plugins-config-volume\") pod \"eb62aecb-6dab-41b6-a34f-1998e886ac98\" (UID: \"eb62aecb-6dab-41b6-a34f-1998e886ac98\") " Jul 29 01:52:32.226934 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.226842 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1-openshift-service-ca\") pod \"24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1\" (UID: \"24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1\") " Jul 29 01:52:32.227244 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.227221 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb62aecb-6dab-41b6-a34f-1998e886ac98-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb62aecb-6dab-41b6-a34f-1998e886ac98" (UID: "eb62aecb-6dab-41b6-a34f-1998e886ac98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:52:32.227288 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.227237 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1" (UID: "24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:52:32.228976 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.228952 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1" (UID: "24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 01:52:32.229064 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.229043 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb62aecb-6dab-41b6-a34f-1998e886ac98-kube-api-access-snhch" (OuterVolumeSpecName: "kube-api-access-snhch") pod "eb62aecb-6dab-41b6-a34f-1998e886ac98" (UID: "eb62aecb-6dab-41b6-a34f-1998e886ac98"). InnerVolumeSpecName "kube-api-access-snhch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:52:32.229104 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.229050 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1-kube-api-access-kgcwt" (OuterVolumeSpecName: "kube-api-access-kgcwt") pod "24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1" (UID: "24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1"). InnerVolumeSpecName "kube-api-access-kgcwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:52:32.328082 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.328056 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1-maas-api-tls\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:52:32.328082 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.328082 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snhch\" (UniqueName: \"kubernetes.io/projected/eb62aecb-6dab-41b6-a34f-1998e886ac98-kube-api-access-snhch\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:52:32.328264 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.328092 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgcwt\" (UniqueName: \"kubernetes.io/projected/24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1-kube-api-access-kgcwt\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:52:32.328264 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.328102 2543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb62aecb-6dab-41b6-a34f-1998e886ac98-plugins-config-volume\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:52:32.328264 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.328113 2543 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1-openshift-service-ca\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:52:32.343741 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.343710 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d75ed66a-24d7-41bf-a620-a7c340bb8595" path="/var/lib/kubelet/pods/d75ed66a-24d7-41bf-a620-a7c340bb8595/volumes" Jul 29 01:52:32.741204 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.741081 2543 generic.go:356] "Generic (PLEG): container finished" podID="eb62aecb-6dab-41b6-a34f-1998e886ac98" containerID="c0e36400a3e3a7f5cdda54edbe8b14225a83d86f4feb72e8f11ec735a2ee0d33" exitCode=0 Jul 29 01:52:32.741204 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.741131 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw" Jul 29 01:52:32.741204 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.741170 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw" event={"ID":"eb62aecb-6dab-41b6-a34f-1998e886ac98","Type":"ContainerDied","Data":"c0e36400a3e3a7f5cdda54edbe8b14225a83d86f4feb72e8f11ec735a2ee0d33"} Jul 29 01:52:32.741485 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.741237 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw" event={"ID":"eb62aecb-6dab-41b6-a34f-1998e886ac98","Type":"ContainerDied","Data":"4035ff2d2bdccd3fbf1b297319e1276cf78d6657264dab6a3daf79676af0fcb0"} Jul 29 01:52:32.741485 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.741268 2543 scope.go:117] "RemoveContainer" containerID="c0e36400a3e3a7f5cdda54edbe8b14225a83d86f4feb72e8f11ec735a2ee0d33" Jul 29 01:52:32.742425 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.742404 2543 generic.go:356] "Generic (PLEG): container finished" podID="24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1" containerID="cf1ead341346840c6b153aaa0783370e45bf078614b2e54de3f0b6e5417186d0" exitCode=0 Jul 29 01:52:32.742519 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.742457 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" event={"ID":"24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1","Type":"ContainerDied","Data":"cf1ead341346840c6b153aaa0783370e45bf078614b2e54de3f0b6e5417186d0"} Jul 29 01:52:32.742519 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.742463 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" Jul 29 01:52:32.742519 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.742498 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl" event={"ID":"24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1","Type":"ContainerDied","Data":"4caad00e579b6b719e872bf2b69bea417bd00594eaca0d115d2086141333d07d"} Jul 29 01:52:32.750011 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.749975 2543 scope.go:117] "RemoveContainer" containerID="c0e36400a3e3a7f5cdda54edbe8b14225a83d86f4feb72e8f11ec735a2ee0d33" Jul 29 01:52:32.750255 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:52:32.750235 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0e36400a3e3a7f5cdda54edbe8b14225a83d86f4feb72e8f11ec735a2ee0d33\": container with ID starting with c0e36400a3e3a7f5cdda54edbe8b14225a83d86f4feb72e8f11ec735a2ee0d33 not found: ID does not exist" containerID="c0e36400a3e3a7f5cdda54edbe8b14225a83d86f4feb72e8f11ec735a2ee0d33" Jul 29 01:52:32.750338 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.750262 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0e36400a3e3a7f5cdda54edbe8b14225a83d86f4feb72e8f11ec735a2ee0d33"} err="failed to get container status \"c0e36400a3e3a7f5cdda54edbe8b14225a83d86f4feb72e8f11ec735a2ee0d33\": rpc error: code = NotFound desc = could not find container \"c0e36400a3e3a7f5cdda54edbe8b14225a83d86f4feb72e8f11ec735a2ee0d33\": container with ID starting with c0e36400a3e3a7f5cdda54edbe8b14225a83d86f4feb72e8f11ec735a2ee0d33 not found: ID does not exist" Jul 29 01:52:32.750338 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.750280 2543 scope.go:117] "RemoveContainer" containerID="cf1ead341346840c6b153aaa0783370e45bf078614b2e54de3f0b6e5417186d0" Jul 29 01:52:32.757762 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.757634 2543 scope.go:117] "RemoveContainer" containerID="cf1ead341346840c6b153aaa0783370e45bf078614b2e54de3f0b6e5417186d0" Jul 29 01:52:32.758058 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:52:32.757962 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf1ead341346840c6b153aaa0783370e45bf078614b2e54de3f0b6e5417186d0\": container with ID starting with cf1ead341346840c6b153aaa0783370e45bf078614b2e54de3f0b6e5417186d0 not found: ID does not exist" containerID="cf1ead341346840c6b153aaa0783370e45bf078614b2e54de3f0b6e5417186d0" Jul 29 01:52:32.758058 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.758012 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1ead341346840c6b153aaa0783370e45bf078614b2e54de3f0b6e5417186d0"} err="failed to get container status \"cf1ead341346840c6b153aaa0783370e45bf078614b2e54de3f0b6e5417186d0\": rpc error: code = NotFound desc = could not find container \"cf1ead341346840c6b153aaa0783370e45bf078614b2e54de3f0b6e5417186d0\": container with ID starting with cf1ead341346840c6b153aaa0783370e45bf078614b2e54de3f0b6e5417186d0 not found: ID does not exist" Jul 29 01:52:32.761473 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.761451 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw"] Jul 29 01:52:32.764651 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.764629 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-5ea1e0-7dd5fbf4b4-tshbw"] Jul 29 01:52:32.771242 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.771223 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl"] Jul 29 01:52:32.775019 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:32.775002 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5ea1e0-6866fd78b-rm8kl"] Jul 29 01:52:34.342961 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:34.342920 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1" path="/var/lib/kubelet/pods/24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1/volumes" Jul 29 01:52:34.343474 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:34.343450 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb62aecb-6dab-41b6-a34f-1998e886ac98" path="/var/lib/kubelet/pods/eb62aecb-6dab-41b6-a34f-1998e886ac98/volumes" Jul 29 01:52:38.489965 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:38.489934 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk"] Jul 29 01:52:38.490368 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:38.490146 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" podUID="93dc3d48-5755-40a3-9d66-a501ef6144ab" containerName="istio-proxy" containerID="cri-o://00be825c4e51c8e87edfb985c600d793fc8bbd88353a646689c814300b1398f5" gracePeriod=30 Jul 29 01:52:40.875556 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:40.875512 2543 patch_prober.go:28] interesting pod/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 01:52:40.876000 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:40.875578 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" podUID="93dc3d48-5755-40a3-9d66-a501ef6144ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 01:52:43.736265 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.736240 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:52:43.800026 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.799956 2543 generic.go:356] "Generic (PLEG): container finished" podID="93dc3d48-5755-40a3-9d66-a501ef6144ab" containerID="00be825c4e51c8e87edfb985c600d793fc8bbd88353a646689c814300b1398f5" exitCode=0 Jul 29 01:52:43.800026 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.800003 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" Jul 29 01:52:43.800197 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.800040 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" event={"ID":"93dc3d48-5755-40a3-9d66-a501ef6144ab","Type":"ContainerDied","Data":"00be825c4e51c8e87edfb985c600d793fc8bbd88353a646689c814300b1398f5"} Jul 29 01:52:43.800197 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.800095 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk" event={"ID":"93dc3d48-5755-40a3-9d66-a501ef6144ab","Type":"ContainerDied","Data":"ffb6064512ea0eb9bb4957c2a3d16006313356e0bcf79f78075105caa9783f02"} Jul 29 01:52:43.800197 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.800124 2543 scope.go:117] "RemoveContainer" containerID="00be825c4e51c8e87edfb985c600d793fc8bbd88353a646689c814300b1398f5" Jul 29 01:52:43.808793 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.808778 2543 scope.go:117] "RemoveContainer" containerID="21879f7446aa5be80d66c1c37068f694d49a28bc8fda159529978231e19d2ecf" Jul 29 01:52:43.816320 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.816302 2543 scope.go:117] "RemoveContainer" containerID="00be825c4e51c8e87edfb985c600d793fc8bbd88353a646689c814300b1398f5" Jul 29 01:52:43.816572 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:52:43.816553 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00be825c4e51c8e87edfb985c600d793fc8bbd88353a646689c814300b1398f5\": container with ID starting with 00be825c4e51c8e87edfb985c600d793fc8bbd88353a646689c814300b1398f5 not found: ID does not exist" containerID="00be825c4e51c8e87edfb985c600d793fc8bbd88353a646689c814300b1398f5" Jul 29 01:52:43.816645 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.816584 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00be825c4e51c8e87edfb985c600d793fc8bbd88353a646689c814300b1398f5"} err="failed to get container status \"00be825c4e51c8e87edfb985c600d793fc8bbd88353a646689c814300b1398f5\": rpc error: code = NotFound desc = could not find container \"00be825c4e51c8e87edfb985c600d793fc8bbd88353a646689c814300b1398f5\": container with ID starting with 00be825c4e51c8e87edfb985c600d793fc8bbd88353a646689c814300b1398f5 not found: ID does not exist" Jul 29 01:52:43.816645 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.816603 2543 scope.go:117] "RemoveContainer" containerID="21879f7446aa5be80d66c1c37068f694d49a28bc8fda159529978231e19d2ecf" Jul 29 01:52:43.816905 ip-10-0-132-136 kubenswrapper[2543]: E0729 01:52:43.816880 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21879f7446aa5be80d66c1c37068f694d49a28bc8fda159529978231e19d2ecf\": container with ID starting with 21879f7446aa5be80d66c1c37068f694d49a28bc8fda159529978231e19d2ecf not found: ID does not exist" containerID="21879f7446aa5be80d66c1c37068f694d49a28bc8fda159529978231e19d2ecf" Jul 29 01:52:43.816955 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.816915 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21879f7446aa5be80d66c1c37068f694d49a28bc8fda159529978231e19d2ecf"} err="failed to get container status \"21879f7446aa5be80d66c1c37068f694d49a28bc8fda159529978231e19d2ecf\": rpc error: code = NotFound desc = could not find container \"21879f7446aa5be80d66c1c37068f694d49a28bc8fda159529978231e19d2ecf\": container with ID starting with 21879f7446aa5be80d66c1c37068f694d49a28bc8fda159529978231e19d2ecf not found: ID does not exist" Jul 29 01:52:43.835141 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.835123 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-envoy\") pod \"93dc3d48-5755-40a3-9d66-a501ef6144ab\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " Jul 29 01:52:43.835230 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.835164 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-workload-socket\") pod \"93dc3d48-5755-40a3-9d66-a501ef6144ab\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " Jul 29 01:52:43.835230 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.835192 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-token\") pod \"93dc3d48-5755-40a3-9d66-a501ef6144ab\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " Jul 29 01:52:43.835312 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.835237 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93dc3d48-5755-40a3-9d66-a501ef6144ab-istiod-ca-cert\") pod \"93dc3d48-5755-40a3-9d66-a501ef6144ab\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " Jul 29 01:52:43.835312 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.835269 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-data\") pod \"93dc3d48-5755-40a3-9d66-a501ef6144ab\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " Jul 29 01:52:43.835388 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.835309 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-podinfo\") pod \"93dc3d48-5755-40a3-9d66-a501ef6144ab\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " Jul 29 01:52:43.835388 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.835345 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwx6d\" (UniqueName: \"kubernetes.io/projected/93dc3d48-5755-40a3-9d66-a501ef6144ab-kube-api-access-mwx6d\") pod \"93dc3d48-5755-40a3-9d66-a501ef6144ab\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " Jul 29 01:52:43.835388 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.835381 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-credential-socket\") pod \"93dc3d48-5755-40a3-9d66-a501ef6144ab\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " Jul 29 01:52:43.835536 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.835396 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "93dc3d48-5755-40a3-9d66-a501ef6144ab" (UID: "93dc3d48-5755-40a3-9d66-a501ef6144ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:52:43.835536 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.835415 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-workload-certs\") pod \"93dc3d48-5755-40a3-9d66-a501ef6144ab\" (UID: \"93dc3d48-5755-40a3-9d66-a501ef6144ab\") " Jul 29 01:52:43.835657 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.835628 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93dc3d48-5755-40a3-9d66-a501ef6144ab-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "93dc3d48-5755-40a3-9d66-a501ef6144ab" (UID: "93dc3d48-5755-40a3-9d66-a501ef6144ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:52:43.835761 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.835736 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-workload-socket\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:52:43.835761 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.835757 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93dc3d48-5755-40a3-9d66-a501ef6144ab-istiod-ca-cert\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:52:43.835927 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.835870 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "93dc3d48-5755-40a3-9d66-a501ef6144ab" (UID: "93dc3d48-5755-40a3-9d66-a501ef6144ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:52:43.836100 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.836074 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "93dc3d48-5755-40a3-9d66-a501ef6144ab" (UID: "93dc3d48-5755-40a3-9d66-a501ef6144ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:52:43.836278 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.836249 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-data" (OuterVolumeSpecName: "istio-data") pod "93dc3d48-5755-40a3-9d66-a501ef6144ab" (UID: "93dc3d48-5755-40a3-9d66-a501ef6144ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:52:43.837416 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.837393 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-token" (OuterVolumeSpecName: "istio-token") pod "93dc3d48-5755-40a3-9d66-a501ef6144ab" (UID: "93dc3d48-5755-40a3-9d66-a501ef6144ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:52:43.837618 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.837598 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "93dc3d48-5755-40a3-9d66-a501ef6144ab" (UID: "93dc3d48-5755-40a3-9d66-a501ef6144ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 01:52:43.837685 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.837667 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93dc3d48-5755-40a3-9d66-a501ef6144ab-kube-api-access-mwx6d" (OuterVolumeSpecName: "kube-api-access-mwx6d") pod "93dc3d48-5755-40a3-9d66-a501ef6144ab" (UID: "93dc3d48-5755-40a3-9d66-a501ef6144ab"). InnerVolumeSpecName "kube-api-access-mwx6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:52:43.837729 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.837683 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "93dc3d48-5755-40a3-9d66-a501ef6144ab" (UID: "93dc3d48-5755-40a3-9d66-a501ef6144ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 01:52:43.937230 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.937194 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-token\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:52:43.937230 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.937220 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-data\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:52:43.937230 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.937228 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-podinfo\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:52:43.937230 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.937237 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwx6d\" (UniqueName: \"kubernetes.io/projected/93dc3d48-5755-40a3-9d66-a501ef6144ab-kube-api-access-mwx6d\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:52:43.937498 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.937247 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-credential-socket\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:52:43.937498 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.937256 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-workload-certs\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:52:43.937498 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:43.937263 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93dc3d48-5755-40a3-9d66-a501ef6144ab-istio-envoy\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:52:44.125906 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:44.125874 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk"] Jul 29 01:52:44.127611 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:44.127593 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-5ea1e0-openshift-default-5fd9f957d4-rd4sk"] Jul 29 01:52:44.343063 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:44.343031 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93dc3d48-5755-40a3-9d66-a501ef6144ab" path="/var/lib/kubelet/pods/93dc3d48-5755-40a3-9d66-a501ef6144ab/volumes" Jul 29 01:52:49.590088 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:49.590055 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq"] Jul 29 01:52:49.590921 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:49.590864 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq" podUID="9b2a5e68-4682-48d5-ac5c-d2e938a08419" containerName="payload-processing" containerID="cri-o://bf1059cd659cb8011103828cdf97b66bd16bc00d469efe93ea9b45142364ffbd" gracePeriod=30 Jul 29 01:52:49.839136 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:49.839101 2543 generic.go:356] "Generic (PLEG): container finished" podID="9b2a5e68-4682-48d5-ac5c-d2e938a08419" containerID="bf1059cd659cb8011103828cdf97b66bd16bc00d469efe93ea9b45142364ffbd" exitCode=0 Jul 29 01:52:49.839292 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:49.839176 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq" event={"ID":"9b2a5e68-4682-48d5-ac5c-d2e938a08419","Type":"ContainerDied","Data":"bf1059cd659cb8011103828cdf97b66bd16bc00d469efe93ea9b45142364ffbd"} Jul 29 01:52:49.839292 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:49.839232 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq" event={"ID":"9b2a5e68-4682-48d5-ac5c-d2e938a08419","Type":"ContainerDied","Data":"b5e1a80a700c2fbdf113edfbe0bb884ee75cea6dc1b21c50b6429b4c107eda37"} Jul 29 01:52:49.839292 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:49.839250 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5e1a80a700c2fbdf113edfbe0bb884ee75cea6dc1b21c50b6429b4c107eda37" Jul 29 01:52:49.856256 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:49.856230 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq" Jul 29 01:52:49.885460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:49.885430 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2a5e68-4682-48d5-ac5c-d2e938a08419-plugins-config-volume\") pod \"9b2a5e68-4682-48d5-ac5c-d2e938a08419\" (UID: \"9b2a5e68-4682-48d5-ac5c-d2e938a08419\") " Jul 29 01:52:49.885584 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:49.885478 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csnpm\" (UniqueName: \"kubernetes.io/projected/9b2a5e68-4682-48d5-ac5c-d2e938a08419-kube-api-access-csnpm\") pod \"9b2a5e68-4682-48d5-ac5c-d2e938a08419\" (UID: \"9b2a5e68-4682-48d5-ac5c-d2e938a08419\") " Jul 29 01:52:49.885729 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:49.885708 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b2a5e68-4682-48d5-ac5c-d2e938a08419-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9b2a5e68-4682-48d5-ac5c-d2e938a08419" (UID: "9b2a5e68-4682-48d5-ac5c-d2e938a08419"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 01:52:49.887590 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:49.887571 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2a5e68-4682-48d5-ac5c-d2e938a08419-kube-api-access-csnpm" (OuterVolumeSpecName: "kube-api-access-csnpm") pod "9b2a5e68-4682-48d5-ac5c-d2e938a08419" (UID: "9b2a5e68-4682-48d5-ac5c-d2e938a08419"). InnerVolumeSpecName "kube-api-access-csnpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 01:52:49.986911 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:49.986876 2543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2a5e68-4682-48d5-ac5c-d2e938a08419-plugins-config-volume\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:52:49.986911 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:49.986904 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csnpm\" (UniqueName: \"kubernetes.io/projected/9b2a5e68-4682-48d5-ac5c-d2e938a08419-kube-api-access-csnpm\") on node \"ip-10-0-132-136.ec2.internal\" DevicePath \"\"" Jul 29 01:52:50.843979 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:50.843948 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq" Jul 29 01:52:50.857945 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:50.857921 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq"] Jul 29 01:52:50.866970 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:50.866944 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-da474e-8475645c54-vkzkq"] Jul 29 01:52:52.343219 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:52:52.343189 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b2a5e68-4682-48d5-ac5c-d2e938a08419" path="/var/lib/kubelet/pods/9b2a5e68-4682-48d5-ac5c-d2e938a08419/volumes" Jul 29 01:53:02.696309 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:02.696273 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-795cff746b-c2pjl_66b5b8b1-f61a-4ff2-8e1a-8f006dc19156/manager/0.log" Jul 29 01:53:02.702877 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:02.702854 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-795cff746b-c2pjl_66b5b8b1-f61a-4ff2-8e1a-8f006dc19156/copy-manifests/0.log" Jul 29 01:53:03.484789 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:03.484760 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754810-7875b_f34540ad-bcb5-4ec5-9f44-e203bb03415e/cleanup/0.log" Jul 29 01:53:03.592580 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:03.592554 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754825-sddpw_e48f5eca-498c-4403-8622-5cc7720b8be5/cleanup/0.log" Jul 29 01:53:06.036425 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:06.036395 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn_d526833f-d86b-457b-bb62-730fc80ba854/istio-proxy/0.log" Jul 29 01:53:06.151975 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:06.151943 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hb7g6_a1ae36f8-1394-42d6-ab5e-855207cbdd69/discovery/0.log" Jul 29 01:53:06.488121 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:06.488091 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lkmf2_e366d26d-8475-439a-a70d-65c9a247b4a6/istio-proxy/1.log" Jul 29 01:53:06.835472 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:06.835441 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d6d7bcf7c-62rxf_e5dfce0d-d035-4da9-92bc-c6247d5b48c8/router/0.log" Jul 29 01:53:07.274095 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:07.273999 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-r5hfh_9334c1ac-c7c7-4414-93d4-142926871eae/main/0.log" Jul 29 01:53:07.491005 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:07.490967 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcgjvf_78537497-df70-464f-b54b-ffdeda1aa8c2/main/0.log" Jul 29 01:53:10.671596 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:10.671563 2543 scope.go:117] "RemoveContainer" containerID="4e6f6243b0488e65f5e9db4ec75392b6f4d648ded0499100ab6a09b780b097e1" Jul 29 01:53:10.679691 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:10.679670 2543 scope.go:117] "RemoveContainer" containerID="bf1059cd659cb8011103828cdf97b66bd16bc00d469efe93ea9b45142364ffbd" Jul 29 01:53:16.433050 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:16.432993 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lf9gq_03f35a91-81af-4c82-976b-50ad3b9d4ea6/global-pull-secret-syncer/0.log" Jul 29 01:53:16.576008 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:16.575972 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j7xsc_0b304c44-8f9d-4830-a2e7-d5559eeb879f/konnectivity-agent/0.log" Jul 29 01:53:16.622871 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:16.622841 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-136.ec2.internal_e1b4121d255f123d5bc2e0fe24413a8b/haproxy/0.log" Jul 29 01:53:24.942697 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:24.942655 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-zt4lx_90ebb0ff-2b60-4c52-97cb-1cf589c21b61/cluster-monitoring-operator/0.log" Jul 29 01:53:25.234899 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:25.234803 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w6wbc_4ce1f418-f28e-4c13-9fe2-e02d0dec2a90/node-exporter/0.log" Jul 29 01:53:25.266234 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:25.266192 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w6wbc_4ce1f418-f28e-4c13-9fe2-e02d0dec2a90/kube-rbac-proxy/0.log" Jul 29 01:53:25.287318 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:25.287302 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w6wbc_4ce1f418-f28e-4c13-9fe2-e02d0dec2a90/init-textfile/0.log" Jul 29 01:53:25.535874 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:25.535783 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-z2x8p_d5fa44cc-c54b-48fe-bcd0-e27c4cb85318/prometheus-operator/0.log" Jul 29 01:53:25.561420 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:25.561388 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-z2x8p_d5fa44cc-c54b-48fe-bcd0-e27c4cb85318/kube-rbac-proxy/0.log" Jul 29 01:53:25.585940 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:25.585915 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-bcsq5_60f6356c-85a2-462a-8855-df307d6093bc/prometheus-operator-admission-webhook/0.log" Jul 29 01:53:26.139649 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.139615 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc"] Jul 29 01:53:26.140058 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.139968 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d512fff-d4f8-4ac0-af92-1023a2dfae3e" containerName="payload-processing" Jul 29 01:53:26.140058 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.139981 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d512fff-d4f8-4ac0-af92-1023a2dfae3e" containerName="payload-processing" Jul 29 01:53:26.140058 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.139990 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a6a9869-1068-4c80-8773-70ebf4f6d1e9" containerName="maas-api" Jul 29 01:53:26.140058 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.139996 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a6a9869-1068-4c80-8773-70ebf4f6d1e9" containerName="maas-api" Jul 29 01:53:26.140058 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140004 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b2a5e68-4682-48d5-ac5c-d2e938a08419" containerName="payload-processing" Jul 29 01:53:26.140058 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140010 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b2a5e68-4682-48d5-ac5c-d2e938a08419" containerName="payload-processing" Jul 29 01:53:26.140058 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140018 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c283ffd5-04ec-4c64-8019-45da4b5aad0b" containerName="maas-api" Jul 29 01:53:26.140058 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140024 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="c283ffd5-04ec-4c64-8019-45da4b5aad0b" containerName="maas-api" Jul 29 01:53:26.140058 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140035 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1" containerName="maas-api" Jul 29 01:53:26.140058 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140041 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1" containerName="maas-api" Jul 29 01:53:26.140058 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140047 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b0743f8-60ec-4f0f-a2a5-98ab5f206db7" containerName="payload-processing" Jul 29 01:53:26.140058 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140053 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b0743f8-60ec-4f0f-a2a5-98ab5f206db7" containerName="payload-processing" Jul 29 01:53:26.140058 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140061 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb62aecb-6dab-41b6-a34f-1998e886ac98" containerName="payload-pre-processing" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140067 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb62aecb-6dab-41b6-a34f-1998e886ac98" containerName="payload-pre-processing" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140074 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93dc3d48-5755-40a3-9d66-a501ef6144ab" containerName="istio-proxy" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140079 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="93dc3d48-5755-40a3-9d66-a501ef6144ab" containerName="istio-proxy" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140085 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93dc3d48-5755-40a3-9d66-a501ef6144ab" containerName="istio-proxy" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140090 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="93dc3d48-5755-40a3-9d66-a501ef6144ab" containerName="istio-proxy" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140101 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a31040a-c8a7-45ac-b508-5cac456360a6" containerName="payload-pre-processing" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140105 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a31040a-c8a7-45ac-b508-5cac456360a6" containerName="payload-pre-processing" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140110 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f6cf9cc-204c-4ac0-8889-81eb506a1375" containerName="payload-processing" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140115 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f6cf9cc-204c-4ac0-8889-81eb506a1375" containerName="payload-processing" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140121 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7" containerName="payload-pre-processing" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140126 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7" containerName="payload-pre-processing" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140131 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb63cf49-dd44-44e4-8564-3be4e2360251" containerName="maas-api" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140136 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb63cf49-dd44-44e4-8564-3be4e2360251" containerName="maas-api" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140182 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="93dc3d48-5755-40a3-9d66-a501ef6144ab" containerName="istio-proxy" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140189 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b2a5e68-4682-48d5-ac5c-d2e938a08419" containerName="payload-processing" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140197 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="24c4f5c6-2bd1-4fd6-9aec-4c8b7aac17f1" containerName="maas-api" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140202 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a31040a-c8a7-45ac-b508-5cac456360a6" containerName="payload-pre-processing" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140209 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b0743f8-60ec-4f0f-a2a5-98ab5f206db7" containerName="payload-processing" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140214 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d512fff-d4f8-4ac0-af92-1023a2dfae3e" containerName="payload-processing" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140221 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f6cf9cc-204c-4ac0-8889-81eb506a1375" containerName="payload-processing" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140226 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a6a9869-1068-4c80-8773-70ebf4f6d1e9" containerName="maas-api" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140231 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="691ae1cc-ab49-4ebb-85ed-3cc73b7e83a7" containerName="payload-pre-processing" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140237 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb62aecb-6dab-41b6-a34f-1998e886ac98" containerName="payload-pre-processing" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140242 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb63cf49-dd44-44e4-8564-3be4e2360251" containerName="maas-api" Jul 29 01:53:26.140460 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.140247 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="c283ffd5-04ec-4c64-8019-45da4b5aad0b" containerName="maas-api" Jul 29 01:53:26.143149 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.143133 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc" Jul 29 01:53:26.145295 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.145267 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4fd22\"/\"openshift-service-ca.crt\"" Jul 29 01:53:26.145430 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.145349 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4fd22\"/\"default-dockercfg-8vnr7\"" Jul 29 01:53:26.145430 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.145383 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4fd22\"/\"kube-root-ca.crt\"" Jul 29 01:53:26.154804 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.154776 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc"] Jul 29 01:53:26.320133 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.320102 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fdef3b98-4b77-4738-b2b3-f8806adbe49a-lib-modules\") pod \"perf-node-gather-daemonset-2cvqc\" (UID: \"fdef3b98-4b77-4738-b2b3-f8806adbe49a\") " pod="openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc" Jul 29 01:53:26.320323 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.320152 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdef3b98-4b77-4738-b2b3-f8806adbe49a-sys\") pod \"perf-node-gather-daemonset-2cvqc\" (UID: \"fdef3b98-4b77-4738-b2b3-f8806adbe49a\") " pod="openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc" Jul 29 01:53:26.320323 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.320264 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fdef3b98-4b77-4738-b2b3-f8806adbe49a-podres\") pod \"perf-node-gather-daemonset-2cvqc\" (UID: \"fdef3b98-4b77-4738-b2b3-f8806adbe49a\") " pod="openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc" Jul 29 01:53:26.320323 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.320294 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6ggl\" (UniqueName: \"kubernetes.io/projected/fdef3b98-4b77-4738-b2b3-f8806adbe49a-kube-api-access-j6ggl\") pod \"perf-node-gather-daemonset-2cvqc\" (UID: \"fdef3b98-4b77-4738-b2b3-f8806adbe49a\") " pod="openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc" Jul 29 01:53:26.320449 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.320326 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fdef3b98-4b77-4738-b2b3-f8806adbe49a-proc\") pod \"perf-node-gather-daemonset-2cvqc\" (UID: \"fdef3b98-4b77-4738-b2b3-f8806adbe49a\") " pod="openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc" Jul 29 01:53:26.421184 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.421094 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fdef3b98-4b77-4738-b2b3-f8806adbe49a-proc\") pod \"perf-node-gather-daemonset-2cvqc\" (UID: \"fdef3b98-4b77-4738-b2b3-f8806adbe49a\") " pod="openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc" Jul 29 01:53:26.421184 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.421139 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fdef3b98-4b77-4738-b2b3-f8806adbe49a-lib-modules\") pod \"perf-node-gather-daemonset-2cvqc\" (UID: \"fdef3b98-4b77-4738-b2b3-f8806adbe49a\") " pod="openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc" Jul 29 01:53:26.421184 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.421180 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdef3b98-4b77-4738-b2b3-f8806adbe49a-sys\") pod \"perf-node-gather-daemonset-2cvqc\" (UID: \"fdef3b98-4b77-4738-b2b3-f8806adbe49a\") " pod="openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc" Jul 29 01:53:26.421445 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.421217 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fdef3b98-4b77-4738-b2b3-f8806adbe49a-proc\") pod \"perf-node-gather-daemonset-2cvqc\" (UID: \"fdef3b98-4b77-4738-b2b3-f8806adbe49a\") " pod="openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc" Jul 29 01:53:26.421445 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.421237 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fdef3b98-4b77-4738-b2b3-f8806adbe49a-podres\") pod \"perf-node-gather-daemonset-2cvqc\" (UID: \"fdef3b98-4b77-4738-b2b3-f8806adbe49a\") " pod="openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc" Jul 29 01:53:26.421445 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.421272 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6ggl\" (UniqueName: \"kubernetes.io/projected/fdef3b98-4b77-4738-b2b3-f8806adbe49a-kube-api-access-j6ggl\") pod \"perf-node-gather-daemonset-2cvqc\" (UID: \"fdef3b98-4b77-4738-b2b3-f8806adbe49a\") " pod="openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc" Jul 29 01:53:26.421445 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.421292 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdef3b98-4b77-4738-b2b3-f8806adbe49a-sys\") pod \"perf-node-gather-daemonset-2cvqc\" (UID: \"fdef3b98-4b77-4738-b2b3-f8806adbe49a\") " pod="openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc" Jul 29 01:53:26.421445 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.421347 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fdef3b98-4b77-4738-b2b3-f8806adbe49a-lib-modules\") pod \"perf-node-gather-daemonset-2cvqc\" (UID: \"fdef3b98-4b77-4738-b2b3-f8806adbe49a\") " pod="openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc" Jul 29 01:53:26.421445 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.421414 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fdef3b98-4b77-4738-b2b3-f8806adbe49a-podres\") pod \"perf-node-gather-daemonset-2cvqc\" (UID: \"fdef3b98-4b77-4738-b2b3-f8806adbe49a\") " pod="openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc" Jul 29 01:53:26.429470 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.429449 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6ggl\" (UniqueName: \"kubernetes.io/projected/fdef3b98-4b77-4738-b2b3-f8806adbe49a-kube-api-access-j6ggl\") pod \"perf-node-gather-daemonset-2cvqc\" (UID: \"fdef3b98-4b77-4738-b2b3-f8806adbe49a\") " pod="openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc" Jul 29 01:53:26.456032 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.456001 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc" Jul 29 01:53:26.585385 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.585359 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc"] Jul 29 01:53:26.587864 ip-10-0-132-136 kubenswrapper[2543]: W0729 01:53:26.587834 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfdef3b98_4b77_4738_b2b3_f8806adbe49a.slice/crio-aae31a9e903c43cb09d4a350434d5bafe8d6e88bf3b6884452f01d57e677a20b WatchSource:0}: Error finding container aae31a9e903c43cb09d4a350434d5bafe8d6e88bf3b6884452f01d57e677a20b: Status 404 returned error can't find the container with id aae31a9e903c43cb09d4a350434d5bafe8d6e88bf3b6884452f01d57e677a20b Jul 29 01:53:26.992048 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:26.991959 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-b7qp4_b8a3faba-c784-4189-9c88-b6743126556a/networking-console-plugin/0.log" Jul 29 01:53:27.016135 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:27.016104 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc" event={"ID":"fdef3b98-4b77-4738-b2b3-f8806adbe49a","Type":"ContainerStarted","Data":"2422e1328524b5e5bf90c2d7d6dcb57129898b262a556f95594fd1b9a18ab1fb"} Jul 29 01:53:27.016297 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:27.016152 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc" event={"ID":"fdef3b98-4b77-4738-b2b3-f8806adbe49a","Type":"ContainerStarted","Data":"aae31a9e903c43cb09d4a350434d5bafe8d6e88bf3b6884452f01d57e677a20b"} Jul 29 01:53:27.016297 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:27.016184 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc" Jul 29 01:53:27.031702 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:27.031659 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc" podStartSLOduration=1.031644869 podStartE2EDuration="1.031644869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 01:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 01:53:27.030133306 +0000 UTC m=+3557.300777614" watchObservedRunningTime="2026-07-29 01:53:27.031644869 +0000 UTC m=+3557.302289175" Jul 29 01:53:27.564157 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:27.564113 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/2.log" Jul 29 01:53:27.572058 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:27.572032 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dl64r_c6ed4d34-e21f-445c-850b-2ee2e32aa6f0/console-operator/3.log" Jul 29 01:53:28.068927 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:28.068891 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-wlvqj_655d6892-5628-44aa-bb78-baa703d432f8/download-server/0.log" Jul 29 01:53:29.736664 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:29.736634 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7jhhc_078d8065-0985-4266-b9ec-e607f7372bb7/dns/0.log" Jul 29 01:53:29.762709 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:29.762685 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7jhhc_078d8065-0985-4266-b9ec-e607f7372bb7/kube-rbac-proxy/0.log" Jul 29 01:53:29.909397 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:29.909371 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-thq5l_5719a514-b8de-4661-ab2a-fe0587726ae8/dns-node-resolver/0.log" Jul 29 01:53:30.518328 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:30.518292 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nqv9s_a2127c7c-06fc-4849-ab98-2975c0862540/node-ca/0.log" Jul 29 01:53:31.496286 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:31.496251 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn_d526833f-d86b-457b-bb62-730fc80ba854/istio-proxy/0.log" Jul 29 01:53:31.760872 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:31.760754 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hb7g6_a1ae36f8-1394-42d6-ab5e-855207cbdd69/discovery/0.log" Jul 29 01:53:31.836781 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:31.836751 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lkmf2_e366d26d-8475-439a-a70d-65c9a247b4a6/istio-proxy/0.log" Jul 29 01:53:31.888418 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:31.888390 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lkmf2_e366d26d-8475-439a-a70d-65c9a247b4a6/istio-proxy/1.log" Jul 29 01:53:31.991931 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:31.991896 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d6d7bcf7c-62rxf_e5dfce0d-d035-4da9-92bc-c6247d5b48c8/router/0.log" Jul 29 01:53:32.793564 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:32.793527 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h6ncm_9ab87c7b-6a40-4383-a3cb-90f40e463040/serve-healthcheck-canary/0.log" Jul 29 01:53:33.031023 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:33.030993 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4fd22/perf-node-gather-daemonset-2cvqc" Jul 29 01:53:33.543140 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:33.543109 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ppxd8_e12d3020-b8bc-478a-979e-db748500760e/kube-rbac-proxy/0.log" Jul 29 01:53:33.568220 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:33.568190 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ppxd8_e12d3020-b8bc-478a-979e-db748500760e/exporter/0.log" Jul 29 01:53:33.593420 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:33.593392 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ppxd8_e12d3020-b8bc-478a-979e-db748500760e/extractor/0.log" Jul 29 01:53:36.671355 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:36.671324 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-795cff746b-c2pjl_66b5b8b1-f61a-4ff2-8e1a-8f006dc19156/manager/0.log" Jul 29 01:53:36.697267 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:36.697239 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-795cff746b-c2pjl_66b5b8b1-f61a-4ff2-8e1a-8f006dc19156/copy-manifests/0.log" Jul 29 01:53:36.811978 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:36.811939 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn_d526833f-d86b-457b-bb62-730fc80ba854/istio-proxy/0.log" Jul 29 01:53:37.064993 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:37.064960 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hb7g6_a1ae36f8-1394-42d6-ab5e-855207cbdd69/discovery/0.log" Jul 29 01:53:37.112116 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:37.112081 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lkmf2_e366d26d-8475-439a-a70d-65c9a247b4a6/istio-proxy/0.log" Jul 29 01:53:37.164038 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:37.164010 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lkmf2_e366d26d-8475-439a-a70d-65c9a247b4a6/istio-proxy/1.log" Jul 29 01:53:37.239278 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:37.239248 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d6d7bcf7c-62rxf_e5dfce0d-d035-4da9-92bc-c6247d5b48c8/router/0.log" Jul 29 01:53:38.163621 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:38.163583 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6c7bdf789-cgxcg_78787cbb-a635-4cad-8b4c-a2e8c434eaae/manager/0.log" Jul 29 01:53:38.652132 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:38.652102 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-qdwzn_d526833f-d86b-457b-bb62-730fc80ba854/istio-proxy/0.log" Jul 29 01:53:38.919016 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:38.918938 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hb7g6_a1ae36f8-1394-42d6-ab5e-855207cbdd69/discovery/0.log" Jul 29 01:53:38.969416 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:38.969382 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lkmf2_e366d26d-8475-439a-a70d-65c9a247b4a6/istio-proxy/0.log" Jul 29 01:53:39.024370 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:39.024338 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lkmf2_e366d26d-8475-439a-a70d-65c9a247b4a6/istio-proxy/1.log" Jul 29 01:53:39.089831 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:39.089789 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d6d7bcf7c-62rxf_e5dfce0d-d035-4da9-92bc-c6247d5b48c8/router/0.log" Jul 29 01:53:42.988432 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:42.988400 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-q9nqv_2d05aed8-edd7-4d87-8c99-f621979c8178/migrator/0.log" Jul 29 01:53:43.015269 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:43.015242 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-q9nqv_2d05aed8-edd7-4d87-8c99-f621979c8178/graceful-termination/0.log" Jul 29 01:53:43.369961 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:43.369929 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq_842fabe6-40df-4c93-bba3-568afdaa041e/kube-storage-version-migrator-operator/1.log" Jul 29 01:53:43.371709 ip-10-0-132-136 kubenswrapper[2543]: I0729 01:53:43.371687 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-mr9tq_842fabe6-40df-4c93-bba3-568afdaa041e/kube-storage-version-migrator-operator/0.log"