Jul 17 17:49:48.282699 ip-10-0-142-155 systemd[1]: Starting Kubernetes Kubelet... Jul 17 17:49:48.739266 ip-10-0-142-155 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 17:49:48.739266 ip-10-0-142-155 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 17 17:49:48.739266 ip-10-0-142-155 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 17:49:48.739266 ip-10-0-142-155 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 17 17:49:48.739266 ip-10-0-142-155 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 17:49:48.742250 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.742089 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 17 17:49:48.747339 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747322 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:49:48.747339 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747338 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:49:48.747410 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747341 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:49:48.747410 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747344 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:49:48.747410 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747347 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:49:48.747410 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747365 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:49:48.747410 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747368 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:49:48.747410 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747371 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:49:48.747410 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747374 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:49:48.747410 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747377 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:49:48.747410 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747380 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:49:48.747410 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747383 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:49:48.747410 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747386 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:49:48.747410 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747389 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:49:48.747410 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747393 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:49:48.747410 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747397 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:49:48.747410 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747400 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:49:48.747410 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747403 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:49:48.747410 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747406 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:49:48.747410 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747408 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:49:48.747410 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747411 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:49:48.747862 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747414 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:49:48.747862 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747417 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:49:48.747862 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747425 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:49:48.747862 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747429 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:49:48.747862 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747432 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:49:48.747862 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747436 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:49:48.747862 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747440 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:49:48.747862 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747443 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:49:48.747862 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747445 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:49:48.747862 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747448 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:49:48.747862 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747450 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:49:48.747862 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747453 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:49:48.747862 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747456 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:49:48.747862 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747458 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:49:48.747862 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747461 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:49:48.747862 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747463 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:49:48.747862 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747466 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:49:48.747862 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747468 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:49:48.747862 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747471 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:49:48.748312 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747473 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:49:48.748312 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747475 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:49:48.748312 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747479 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:49:48.748312 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747482 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:49:48.748312 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747484 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:49:48.748312 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747486 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:49:48.748312 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747489 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:49:48.748312 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747497 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:49:48.748312 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747500 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:49:48.748312 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747502 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:49:48.748312 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747504 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:49:48.748312 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747507 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:49:48.748312 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747509 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:49:48.748312 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747512 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:49:48.748312 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747515 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:49:48.748312 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747517 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:49:48.748312 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747520 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:49:48.748312 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747523 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:49:48.748312 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747525 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:49:48.748312 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747528 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:49:48.748815 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747530 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:49:48.748815 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747533 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:49:48.748815 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747535 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:49:48.748815 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747538 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:49:48.748815 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747540 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:49:48.748815 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747543 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:49:48.748815 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747545 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:49:48.748815 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747548 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:49:48.748815 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747550 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:49:48.748815 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747553 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:49:48.748815 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747555 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:49:48.748815 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747558 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:49:48.748815 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747561 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:49:48.748815 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747564 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:49:48.748815 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747567 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:49:48.748815 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747570 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:49:48.748815 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747573 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:49:48.748815 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747575 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:49:48.748815 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747578 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:49:48.749261 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747580 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:49:48.749261 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747582 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:49:48.749261 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747585 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:49:48.749261 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747588 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:49:48.749261 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747592 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:49:48.749261 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747594 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:49:48.749261 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747598 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:49:48.749261 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747601 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:49:48.749261 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747603 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:49:48.749261 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747607 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:49:48.749261 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747609 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:49:48.749261 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747611 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:49:48.749261 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747614 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:49:48.749261 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747616 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:49:48.749261 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747619 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:49:48.749261 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747621 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:49:48.749261 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747624 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:49:48.749261 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747626 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:49:48.749261 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747629 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:49:48.749261 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.747631 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:49:48.749261 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748387 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:49:48.749782 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748395 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:49:48.749782 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748400 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:49:48.749782 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748404 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:49:48.749782 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748408 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:49:48.749782 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748411 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:49:48.749782 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748413 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:49:48.749782 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748416 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:49:48.749782 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748419 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:49:48.749782 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748422 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:49:48.749782 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748425 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:49:48.749782 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748427 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:49:48.749782 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748429 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:49:48.749782 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748432 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:49:48.749782 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748434 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:49:48.749782 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748438 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:49:48.749782 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748441 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:49:48.749782 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748444 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:49:48.749782 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748447 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:49:48.749782 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748451 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:49:48.750260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748453 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:49:48.750260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748457 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:49:48.750260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748460 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:49:48.750260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748463 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:49:48.750260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748465 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:49:48.750260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748468 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:49:48.750260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748470 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:49:48.750260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748473 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:49:48.750260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748476 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:49:48.750260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748478 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:49:48.750260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748480 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:49:48.750260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748483 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:49:48.750260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748485 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:49:48.750260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748488 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:49:48.750260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748490 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:49:48.750260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748493 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:49:48.750260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748495 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:49:48.750260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748498 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:49:48.750260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748500 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:49:48.750260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748502 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:49:48.750766 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748505 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:49:48.750766 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748507 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:49:48.750766 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748510 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:49:48.750766 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748513 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:49:48.750766 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748515 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:49:48.750766 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748518 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:49:48.750766 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748520 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:49:48.750766 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748523 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:49:48.750766 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748526 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:49:48.750766 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748528 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:49:48.750766 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748531 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:49:48.750766 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748534 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:49:48.750766 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748537 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:49:48.750766 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748539 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:49:48.750766 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748542 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:49:48.750766 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748544 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:49:48.750766 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748547 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:49:48.750766 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748550 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:49:48.750766 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748552 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:49:48.750766 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748555 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:49:48.751259 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748557 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:49:48.751259 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748560 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:49:48.751259 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748563 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:49:48.751259 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748565 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:49:48.751259 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748568 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:49:48.751259 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748570 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:49:48.751259 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748573 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:49:48.751259 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748575 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:49:48.751259 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748578 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:49:48.751259 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748580 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:49:48.751259 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748583 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:49:48.751259 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748585 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:49:48.751259 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748588 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:49:48.751259 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748590 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:49:48.751259 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748592 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:49:48.751259 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748595 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:49:48.751259 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748598 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:49:48.751259 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748600 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:49:48.751259 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748602 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:49:48.751259 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748606 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:49:48.751754 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748608 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:49:48.751754 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748611 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:49:48.751754 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748613 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:49:48.751754 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748616 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:49:48.751754 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748619 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:49:48.751754 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748621 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:49:48.751754 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748624 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:49:48.751754 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748626 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:49:48.751754 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748628 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:49:48.751754 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748632 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:49:48.751754 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748635 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:49:48.751754 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748638 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:49:48.751754 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748641 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:49:48.751754 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748644 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:49:48.751754 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748646 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:49:48.751754 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748649 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:49:48.751754 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748651 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:49:48.751754 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748654 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:49:48.751754 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.748656 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:49:48.752209 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749037 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:49:48.752209 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749043 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:49:48.752209 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749046 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:49:48.752209 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749049 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:49:48.752209 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749052 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:49:48.752209 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749055 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:49:48.752209 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749058 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:49:48.752209 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749061 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:49:48.752209 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749070 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:49:48.752209 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749073 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:49:48.752209 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749075 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:49:48.752209 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749078 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:49:48.752209 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749081 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:49:48.752209 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749083 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:49:48.752209 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749089 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:49:48.752209 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749092 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:49:48.752209 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749095 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:49:48.752209 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749097 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:49:48.752209 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749100 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:49:48.752703 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749102 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:49:48.752703 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749105 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:49:48.752703 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749107 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:49:48.752703 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749110 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:49:48.752703 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749112 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:49:48.752703 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749115 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:49:48.752703 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749117 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:49:48.752703 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749120 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:49:48.752703 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749122 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:49:48.752703 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749124 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:49:48.752703 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749127 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:49:48.752703 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749129 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:49:48.752703 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749132 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:49:48.752703 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749135 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:49:48.752703 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749137 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:49:48.752703 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749139 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:49:48.752703 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749142 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:49:48.752703 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749145 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:49:48.752703 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749147 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:49:48.753159 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749149 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:49:48.753159 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749152 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:49:48.753159 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749154 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:49:48.753159 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749160 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:49:48.753159 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749162 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:49:48.753159 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749165 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:49:48.753159 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749167 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:49:48.753159 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749170 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:49:48.753159 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749173 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:49:48.753159 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749180 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:49:48.753159 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749183 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:49:48.753159 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749185 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:49:48.753159 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749188 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:49:48.753159 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749190 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:49:48.753159 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749193 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:49:48.753159 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749195 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:49:48.753159 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749198 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:49:48.753159 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749200 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:49:48.753159 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749203 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:49:48.753159 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749205 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:49:48.753675 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749208 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:49:48.753675 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749210 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:49:48.753675 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749215 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:49:48.753675 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749218 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:49:48.753675 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749221 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:49:48.753675 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749224 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:49:48.753675 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749227 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:49:48.753675 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749229 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:49:48.753675 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749231 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:49:48.753675 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749234 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:49:48.753675 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749236 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:49:48.753675 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749239 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:49:48.753675 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749241 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:49:48.753675 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749244 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:49:48.753675 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749246 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:49:48.753675 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749249 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:49:48.753675 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749252 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:49:48.753675 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749255 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:49:48.753675 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749258 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:49:48.754165 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749260 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:49:48.754165 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749267 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:49:48.754165 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749271 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:49:48.754165 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749274 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:49:48.754165 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749277 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:49:48.754165 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749280 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:49:48.754165 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749282 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:49:48.754165 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749284 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:49:48.754165 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749287 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:49:48.754165 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749289 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:49:48.754165 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749292 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:49:48.754165 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749294 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:49:48.754165 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749298 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:49:48.754165 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749309 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:49:48.754165 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749312 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:49:48.754165 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749315 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:49:48.754165 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749318 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:49:48.754165 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749321 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:49:48.754165 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749323 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:49:48.754165 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749325 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:49:48.754165 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749328 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.749330 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749414 2574 flags.go:64] FLAG: --address="0.0.0.0" Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749422 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749437 2574 flags.go:64] FLAG: --anonymous-auth="true" Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749441 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749448 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749452 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749457 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749461 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749465 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749468 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749472 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749476 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749479 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749482 2574 flags.go:64] FLAG: --cgroup-root="" Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749485 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749488 2574 flags.go:64] FLAG: --client-ca-file="" Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749491 2574 flags.go:64] FLAG: --cloud-provider="external" Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749494 2574 flags.go:64] FLAG: --cluster-dns="[]" Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749498 2574 flags.go:64] FLAG: --cluster-domain="" Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749501 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749504 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749507 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 17 17:49:48.754700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749510 2574 flags.go:64] FLAG: --container-log-max-files="5" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749514 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749517 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749520 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749523 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749526 2574 flags.go:64] FLAG: --contention-profiling="false" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749529 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749531 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749534 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749537 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749541 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749544 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749547 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749549 2574 flags.go:64] FLAG: --enable-load-reader="false" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749552 2574 flags.go:64] FLAG: --enable-server="true" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749555 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749559 2574 flags.go:64] FLAG: --event-burst="100" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749562 2574 flags.go:64] FLAG: --event-qps="50" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749565 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749568 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749572 2574 flags.go:64] FLAG: --eviction-hard="" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749575 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749579 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749582 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749585 2574 flags.go:64] FLAG: --eviction-soft="" Jul 17 17:49:48.755298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749588 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749591 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749593 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749596 2574 flags.go:64] FLAG: --experimental-mounter-path="" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749599 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749602 2574 flags.go:64] FLAG: --fail-swap-on="true" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749604 2574 flags.go:64] FLAG: --feature-gates="" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749608 2574 flags.go:64] FLAG: --file-check-frequency="20s" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749611 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749614 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749617 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749619 2574 flags.go:64] FLAG: --healthz-port="10248" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749623 2574 flags.go:64] FLAG: --help="false" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749626 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-142-155.ec2.internal" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749628 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749631 2574 flags.go:64] FLAG: --http-check-frequency="20s" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749634 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749637 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749641 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749643 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749646 2574 flags.go:64] FLAG: --image-service-endpoint="" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749649 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749651 2574 flags.go:64] FLAG: --kube-api-burst="100" Jul 17 17:49:48.755907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749654 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749657 2574 flags.go:64] FLAG: --kube-api-qps="50" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749660 2574 flags.go:64] FLAG: --kube-reserved="" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749663 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749666 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749669 2574 flags.go:64] FLAG: --kubelet-cgroups="" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749672 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749675 2574 flags.go:64] FLAG: --lock-file="" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749678 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749680 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749683 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749689 2574 flags.go:64] FLAG: --log-json-split-stream="false" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749691 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749695 2574 flags.go:64] FLAG: --log-text-split-stream="false" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749697 2574 flags.go:64] FLAG: --logging-format="text" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749700 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749703 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749706 2574 flags.go:64] FLAG: --manifest-url="" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749709 2574 flags.go:64] FLAG: --manifest-url-header="" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749713 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749716 2574 flags.go:64] FLAG: --max-open-files="1000000" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749720 2574 flags.go:64] FLAG: --max-pods="110" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749723 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749726 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749729 2574 flags.go:64] FLAG: --memory-manager-policy="None" Jul 17 17:49:48.756498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749732 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749734 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749737 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749740 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749747 2574 flags.go:64] FLAG: --node-status-max-images="50" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749750 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749753 2574 flags.go:64] FLAG: --oom-score-adj="-999" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749755 2574 flags.go:64] FLAG: --pod-cidr="" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749758 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749763 2574 flags.go:64] FLAG: --pod-manifest-path="" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749766 2574 flags.go:64] FLAG: --pod-max-pids="-1" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749769 2574 flags.go:64] FLAG: --pods-per-core="0" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749772 2574 flags.go:64] FLAG: --port="10250" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749775 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749780 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08e134d604f353192" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749783 2574 flags.go:64] FLAG: --qos-reserved="" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749786 2574 flags.go:64] FLAG: --read-only-port="10255" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749788 2574 flags.go:64] FLAG: --register-node="true" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749792 2574 flags.go:64] FLAG: --register-with-taints="" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749795 2574 flags.go:64] FLAG: --registry-burst="10" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749798 2574 flags.go:64] FLAG: --registry-qps="5" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749801 2574 flags.go:64] FLAG: --reserved-cpus="" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749804 2574 flags.go:64] FLAG: --reserved-memory="" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749807 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749810 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 17 17:49:48.757101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749813 2574 flags.go:64] FLAG: --rotate-certificates="false" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749816 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749819 2574 flags.go:64] FLAG: --runonce="false" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749822 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749825 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749827 2574 flags.go:64] FLAG: --seccomp-default="false" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749830 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749833 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749835 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749838 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749841 2574 flags.go:64] FLAG: --storage-driver-password="root" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749844 2574 flags.go:64] FLAG: --storage-driver-secure="false" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749849 2574 flags.go:64] FLAG: --storage-driver-table="stats" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749851 2574 flags.go:64] FLAG: --storage-driver-user="root" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749854 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749857 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749860 2574 flags.go:64] FLAG: --system-cgroups="" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749862 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749868 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749871 2574 flags.go:64] FLAG: --tls-cert-file="" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749874 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749880 2574 flags.go:64] FLAG: --tls-min-version="" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749883 2574 flags.go:64] FLAG: --tls-private-key-file="" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749886 2574 flags.go:64] FLAG: --topology-manager-policy="none" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749888 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 17 17:49:48.757758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749891 2574 flags.go:64] FLAG: --topology-manager-scope="container" Jul 17 17:49:48.758377 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749894 2574 flags.go:64] FLAG: --v="2" Jul 17 17:49:48.758377 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749898 2574 flags.go:64] FLAG: --version="false" Jul 17 17:49:48.758377 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749902 2574 flags.go:64] FLAG: --vmodule="" Jul 17 17:49:48.758377 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749906 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 17 17:49:48.758377 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.749909 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 17 17:49:48.758377 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750040 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:49:48.758377 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750044 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:49:48.758377 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750047 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:49:48.758377 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750050 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:49:48.758377 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750053 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:49:48.758377 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750056 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:49:48.758377 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750059 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:49:48.758377 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750062 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:49:48.758377 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750064 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:49:48.758377 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750066 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:49:48.758377 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750069 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:49:48.758377 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750071 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:49:48.758377 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750074 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:49:48.758377 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750079 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:49:48.758377 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750082 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:49:48.758902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750085 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:49:48.758902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750088 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:49:48.758902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750091 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:49:48.758902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750094 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:49:48.758902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750097 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:49:48.758902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750100 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:49:48.758902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750102 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:49:48.758902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750107 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:49:48.758902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750110 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:49:48.758902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750113 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:49:48.758902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750115 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:49:48.758902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750118 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:49:48.758902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750121 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:49:48.758902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750124 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:49:48.758902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750126 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:49:48.758902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750129 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:49:48.758902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750131 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:49:48.758902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750133 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:49:48.758902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750136 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:49:48.758902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750139 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:49:48.759530 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750141 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:49:48.759530 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750144 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:49:48.759530 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750146 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:49:48.759530 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750148 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:49:48.759530 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750151 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:49:48.759530 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750154 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:49:48.759530 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750157 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:49:48.759530 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750160 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:49:48.759530 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750163 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:49:48.759530 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750165 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:49:48.759530 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750169 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:49:48.759530 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750172 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:49:48.759530 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750174 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:49:48.759530 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750177 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:49:48.759530 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750179 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:49:48.759530 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750182 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:49:48.759530 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750184 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:49:48.759530 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750186 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:49:48.759530 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750189 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:49:48.760006 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750192 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:49:48.760006 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750196 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:49:48.760006 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750198 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:49:48.760006 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750201 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:49:48.760006 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750203 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:49:48.760006 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750206 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:49:48.760006 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750208 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:49:48.760006 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750211 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:49:48.760006 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750213 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:49:48.760006 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750216 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:49:48.760006 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750218 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:49:48.760006 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750221 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:49:48.760006 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750223 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:49:48.760006 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750226 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:49:48.760006 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750228 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:49:48.760006 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750230 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:49:48.760006 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750233 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:49:48.760006 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750235 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:49:48.760006 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750238 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:49:48.760006 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750240 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:49:48.760006 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750243 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:49:48.760626 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750245 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:49:48.760626 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750247 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:49:48.760626 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750251 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:49:48.760626 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750253 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:49:48.760626 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750256 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:49:48.760626 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750258 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:49:48.760626 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750260 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:49:48.760626 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750263 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:49:48.760626 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750266 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:49:48.760626 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750268 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:49:48.760626 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750271 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:49:48.760626 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750275 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:49:48.760626 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750277 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:49:48.760626 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750280 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:49:48.760626 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750283 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:49:48.760626 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750285 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:49:48.760626 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750288 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:49:48.760626 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750292 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:49:48.760626 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750295 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:49:48.761089 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750298 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:49:48.761089 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750301 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:49:48.761089 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750303 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:49:48.761089 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750306 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:49:48.761089 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.750308 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:49:48.761089 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.751083 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 17:49:48.761089 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.758550 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 17 17:49:48.761089 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.758568 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 17 17:49:48.761089 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.758590 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 17 17:49:48.761089 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.758596 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 17 17:49:48.761089 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758672 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:49:48.761089 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758678 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:49:48.761089 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758681 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:49:48.761089 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758685 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:49:48.761089 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758688 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:49:48.761480 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758691 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:49:48.761480 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758694 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:49:48.761480 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758697 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:49:48.761480 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758700 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:49:48.761480 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758702 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:49:48.761480 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758705 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:49:48.761480 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758708 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:49:48.761480 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758711 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:49:48.761480 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758713 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:49:48.761480 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758716 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:49:48.761480 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758718 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:49:48.761480 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758721 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:49:48.761480 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758723 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:49:48.761480 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758726 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:49:48.761480 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758728 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:49:48.761480 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758730 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:49:48.761480 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758733 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:49:48.761480 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758735 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:49:48.761480 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758738 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:49:48.761979 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758740 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:49:48.761979 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758743 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:49:48.761979 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758746 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:49:48.761979 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758748 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:49:48.761979 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758751 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:49:48.761979 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758755 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:49:48.761979 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758757 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:49:48.761979 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758760 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:49:48.761979 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758762 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:49:48.761979 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758765 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:49:48.761979 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758767 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:49:48.761979 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758770 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:49:48.761979 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758772 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:49:48.761979 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758774 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:49:48.761979 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758777 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:49:48.761979 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758779 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:49:48.761979 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758782 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:49:48.761979 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758784 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:49:48.761979 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758787 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:49:48.761979 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758789 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:49:48.762518 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758792 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:49:48.762518 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758794 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:49:48.762518 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758797 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:49:48.762518 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758799 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:49:48.762518 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758802 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:49:48.762518 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758804 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:49:48.762518 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758806 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:49:48.762518 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758809 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:49:48.762518 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758811 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:49:48.762518 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758814 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:49:48.762518 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758816 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:49:48.762518 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758819 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:49:48.762518 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758821 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:49:48.762518 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758825 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:49:48.762518 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758830 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:49:48.762518 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758834 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:49:48.762518 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758837 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:49:48.762518 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758840 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:49:48.762518 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758844 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:49:48.762518 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758847 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:49:48.763020 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758849 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:49:48.763020 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758852 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:49:48.763020 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758856 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:49:48.763020 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758860 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:49:48.763020 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758863 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:49:48.763020 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758866 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:49:48.763020 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758869 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:49:48.763020 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758872 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:49:48.763020 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758875 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:49:48.763020 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758877 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:49:48.763020 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758880 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:49:48.763020 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758883 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:49:48.763020 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758885 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:49:48.763020 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758888 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:49:48.763020 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758890 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:49:48.763020 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758893 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:49:48.763020 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758896 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:49:48.763020 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758898 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:49:48.763020 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758901 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:49:48.763504 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758903 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:49:48.763504 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758906 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:49:48.763504 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758909 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:49:48.763504 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758911 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:49:48.763504 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758914 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:49:48.763504 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758916 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:49:48.763504 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758919 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:49:48.763504 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758922 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:49:48.763504 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758925 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:49:48.763504 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758927 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:49:48.763504 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758930 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:49:48.763504 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758933 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:49:48.763504 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758936 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:49:48.763504 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758939 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:49:48.763504 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758941 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:49:48.763504 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.758944 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:49:48.763902 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.758949 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 17:49:48.763902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759064 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 17:49:48.763902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759068 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 17:49:48.763902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759071 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 17:49:48.763902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759074 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 17:49:48.763902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759077 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 17:49:48.763902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759080 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 17:49:48.763902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759083 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 17:49:48.763902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759086 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 17:49:48.763902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759089 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 17:49:48.763902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759091 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 17:49:48.763902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759095 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 17:49:48.763902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759099 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 17:49:48.763902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759102 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 17:49:48.763902 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759105 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 17:49:48.764276 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759109 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 17:49:48.764276 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759112 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 17:49:48.764276 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759115 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 17:49:48.764276 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759118 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 17:49:48.764276 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759120 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 17:49:48.764276 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759123 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 17:49:48.764276 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759125 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 17:49:48.764276 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759127 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 17:49:48.764276 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759130 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 17:49:48.764276 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759133 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 17:49:48.764276 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759135 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 17:49:48.764276 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759138 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 17:49:48.764276 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759141 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 17:49:48.764276 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759143 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 17:49:48.764276 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759147 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 17:49:48.764276 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759149 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 17:49:48.764276 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759152 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 17:49:48.764276 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759155 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 17:49:48.764276 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759157 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 17:49:48.764765 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759160 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 17:49:48.764765 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759162 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 17:49:48.764765 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759165 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 17:49:48.764765 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759167 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 17:49:48.764765 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759169 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 17:49:48.764765 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759172 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 17:49:48.764765 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759175 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 17:49:48.764765 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759177 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 17:49:48.764765 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759180 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 17:49:48.764765 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759182 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 17:49:48.764765 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759185 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 17:49:48.764765 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759187 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 17:49:48.764765 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759190 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 17:49:48.764765 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759192 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 17:49:48.764765 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759194 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 17:49:48.764765 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759197 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 17:49:48.764765 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759199 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 17:49:48.764765 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759202 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 17:49:48.764765 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759204 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 17:49:48.764765 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759206 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 17:49:48.765249 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759209 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 17:49:48.765249 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759212 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 17 17:49:48.765249 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759214 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 17:49:48.765249 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759217 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 17:49:48.765249 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759220 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 17:49:48.765249 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759222 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 17:49:48.765249 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759224 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 17:49:48.765249 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759227 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 17:49:48.765249 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759230 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 17:49:48.765249 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759233 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 17:49:48.765249 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759237 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 17:49:48.765249 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759240 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 17:49:48.765249 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759242 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 17:49:48.765249 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759245 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 17:49:48.765249 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759247 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 17:49:48.765249 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759250 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 17:49:48.765249 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759252 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 17:49:48.765249 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759254 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 17:49:48.765249 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759257 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 17:49:48.765249 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759259 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 17:49:48.765763 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759262 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 17:49:48.765763 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759264 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 17:49:48.765763 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759266 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 17:49:48.765763 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759269 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 17:49:48.765763 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759271 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 17:49:48.765763 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759274 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 17:49:48.765763 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759276 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 17:49:48.765763 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759279 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 17:49:48.765763 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759281 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 17:49:48.765763 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759283 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 17:49:48.765763 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759286 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 17:49:48.765763 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759288 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 17:49:48.765763 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759291 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 17:49:48.765763 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759293 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 17:49:48.765763 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759296 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 17:49:48.765763 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759298 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 17:49:48.765763 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759301 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 17:49:48.765763 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759304 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 17:49:48.765763 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759306 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 17:49:48.765763 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759309 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 17:49:48.766260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759311 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 17:49:48.766260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759314 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 17:49:48.766260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759317 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 17:49:48.766260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759319 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 17:49:48.766260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759321 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 17:49:48.766260 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:48.759324 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 17:49:48.766260 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.759329 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 17:49:48.766260 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.759486 2574 server.go:962] "Client rotation is on, will bootstrap in background" Jul 17 17:49:48.766260 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.761988 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 17 17:49:48.766260 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.763774 2574 server.go:1019] "Starting client certificate rotation" Jul 17 17:49:48.766260 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.763879 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 17:49:48.766260 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.763917 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 17:49:48.766650 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.766635 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 17:49:48.772069 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.772039 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 17:49:48.787824 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.787802 2574 log.go:25] "Validated CRI v1 runtime API" Jul 17 17:49:48.790319 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.790305 2574 log.go:25] "Validated CRI v1 image API" Jul 17 17:49:48.791963 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.791945 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 17 17:49:48.796279 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.796247 2574 fs.go:135] Filesystem UUIDs: map[480b79b8-b48a-4a1b-a9c4-e4ada086507a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 94db6907-b1ff-4e7e-9ac9-7133dcfb7bd4:/dev/nvme0n1p4] Jul 17 17:49:48.796378 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.796271 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 17 17:49:48.800009 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.799989 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 17:49:48.801207 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.801086 2574 manager.go:216] Machine: {Timestamp:2026-07-17 17:49:48.800589907 +0000 UTC m=+0.388542588 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100931 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e727037f7debc5bb6c7de412ae78d SystemUUID:ec2e7270-37f7-debc-5bb6-c7de412ae78d BootID:730c74b5-9dde-49a0-accc-c2f8fcd3157c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8c:84:6b:ae:97 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8c:84:6b:ae:97 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:4a:e6:30:c7:f2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 17 17:49:48.801207 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.801194 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 17 17:49:48.801410 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.801323 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 17 17:49:48.804268 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.804237 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 17 17:49:48.804464 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.804272 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-155.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 17 17:49:48.805179 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.805168 2574 topology_manager.go:138] "Creating topology manager with none policy" Jul 17 17:49:48.805213 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.805182 2574 container_manager_linux.go:309] "Creating device plugin manager" Jul 17 17:49:48.805213 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.805194 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 17:49:48.805271 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.805225 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 17:49:48.807116 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.807105 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 17 17:49:48.811794 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.811781 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 17 17:49:48.811922 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.811912 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 17 17:49:48.811963 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.811943 2574 kubelet.go:486] "Attempting to sync node with API server" Jul 17 17:49:48.811963 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.811955 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 17 17:49:48.812025 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.811970 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 17 17:49:48.812025 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.811981 2574 kubelet.go:398] "Adding apiserver pod source" Jul 17 17:49:48.812025 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.811992 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 17 17:49:48.812347 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.812336 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 17:49:48.812404 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.812368 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 17:49:48.814737 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.814714 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 17 17:49:48.816662 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.816647 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 17 17:49:48.816727 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.816678 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 17 17:49:48.817508 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.817497 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 17 17:49:48.817570 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.817514 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 17 17:49:48.817570 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.817520 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 17 17:49:48.817570 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.817525 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 17 17:49:48.817570 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.817535 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 17 17:49:48.817570 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.817540 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 17 17:49:48.817570 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.817548 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 17 17:49:48.817570 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.817553 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 17 17:49:48.817570 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.817561 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 17 17:49:48.817570 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.817566 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 17 17:49:48.817570 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.817574 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 17 17:49:48.817820 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.817580 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 17 17:49:48.819122 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.819112 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 17 17:49:48.819122 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.819123 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 17 17:49:48.820478 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.820457 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vknwm" Jul 17 17:49:48.822502 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.822486 2574 server.go:1268] "Started kubelet" Jul 17 17:49:48.822605 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.822568 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 17 17:49:48.822683 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.822570 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 17 17:49:48.823412 ip-10-0-142-155 systemd[1]: Started Kubernetes Kubelet. Jul 17 17:49:48.826001 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:48.825970 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 17 17:49:48.826089 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.826025 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-155.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 17 17:49:48.826187 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:48.826164 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-155.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 17 17:49:48.826255 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.826188 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 17 17:49:48.827665 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.827645 2574 server.go:310] "Adding debug handlers to kubelet server" Jul 17 17:49:48.827747 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:48.827711 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 17 17:49:48.828326 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.828306 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 17 17:49:48.830506 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.830487 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vknwm" Jul 17 17:49:48.833689 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:48.832558 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-155.ec2.internal.18c32506dfbb007b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-155.ec2.internal,UID:ip-10-0-142-155.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-155.ec2.internal,},FirstTimestamp:2026-07-17 17:49:48.822454395 +0000 UTC m=+0.410407075,LastTimestamp:2026-07-17 17:49:48.822454395 +0000 UTC m=+0.410407075,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-155.ec2.internal,}" Jul 17 17:49:48.838097 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.838082 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 17 17:49:48.838188 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.838163 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 17 17:49:48.838408 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:48.838386 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-155.ec2.internal\" not found" Jul 17 17:49:48.839087 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.839073 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 17 17:49:48.839199 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.839091 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 17 17:49:48.839199 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.839201 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 17 17:49:48.839347 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.839114 2574 factory.go:55] Registering systemd factory Jul 17 17:49:48.839347 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.839275 2574 factory.go:223] Registration of the systemd container factory successfully Jul 17 17:49:48.839347 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.839330 2574 reconstruct.go:97] "Volume reconstruction finished" Jul 17 17:49:48.839347 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.839339 2574 reconciler.go:29] "Reconciler: start to sync state" Jul 17 17:49:48.839640 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.839608 2574 factory.go:153] Registering CRI-O factory Jul 17 17:49:48.839640 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.839641 2574 factory.go:223] Registration of the crio container factory successfully Jul 17 17:49:48.839781 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.839693 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 17 17:49:48.839781 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.839714 2574 factory.go:103] Registering Raw factory Jul 17 17:49:48.839781 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.839724 2574 manager.go:1196] Started watching for new ooms in manager Jul 17 17:49:48.840342 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.840318 2574 manager.go:349] Starting recovery of all containers Jul 17 17:49:48.844891 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.844870 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 17:49:48.849949 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.849820 2574 manager.go:354] Recovery completed Jul 17 17:49:48.851858 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:48.851832 2574 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-155.ec2.internal\" not found" node="ip-10-0-142-155.ec2.internal" Jul 17 17:49:48.853989 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.853977 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:49:48.855785 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.855770 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-155.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:49:48.855857 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.855800 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-155.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:49:48.855857 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.855811 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-155.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:49:48.856727 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.856715 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 17 17:49:48.856727 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.856725 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:49:48.856806 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.856726 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 17 17:49:48.856806 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.856764 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 17 17:49:48.857408 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.857394 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-155.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:49:48.857456 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.857420 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-155.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:49:48.857456 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.857429 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-155.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:49:48.858831 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.858819 2574 policy_none.go:49] "None policy: Start" Jul 17 17:49:48.858865 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.858835 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 17 17:49:48.858865 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.858844 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 17 17:49:48.860723 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.860684 2574 policy_none.go:47] "Start" Jul 17 17:49:48.916690 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.915163 2574 manager.go:337] "Starting Device Plugin manager" Jul 17 17:49:48.916690 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:48.915200 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 17 17:49:48.916690 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.915210 2574 server.go:87] "Starting device plugin registration server" Jul 17 17:49:48.916690 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.915494 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 17 17:49:48.916690 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.915507 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 17 17:49:48.916690 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.915604 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 17 17:49:48.916690 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.915701 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 17 17:49:48.916690 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.915711 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 17 17:49:48.916690 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:48.916302 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 17 17:49:48.916690 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:48.916378 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-155.ec2.internal\" not found" Jul 17 17:49:48.977406 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.977371 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 17 17:49:48.978748 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.978732 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 17 17:49:48.978844 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.978778 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 17 17:49:48.978844 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.978811 2574 kubelet.go:2448] "Starting kubelet main sync loop" Jul 17 17:49:48.978937 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:48.978854 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 17 17:49:48.981536 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:48.981510 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 17:49:49.015922 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.015848 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:49:49.016891 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.016874 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-155.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:49:49.016963 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.016905 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-155.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:49:49.016963 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.016917 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-155.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:49:49.016963 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.016941 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-155.ec2.internal" Jul 17 17:49:49.028005 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.027984 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-155.ec2.internal" Jul 17 17:49:49.028058 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:49.028008 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-155.ec2.internal\": node \"ip-10-0-142-155.ec2.internal\" not found" Jul 17 17:49:49.043392 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:49.043346 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-155.ec2.internal\" not found" Jul 17 17:49:49.079831 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.079770 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-155.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-155.ec2.internal"] Jul 17 17:49:49.079998 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.079872 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:49:49.080880 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.080863 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-155.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:49:49.080970 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.080897 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-155.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:49:49.080970 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.080908 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-155.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:49:49.083294 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.083281 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:49:49.083427 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.083414 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-155.ec2.internal" Jul 17 17:49:49.083471 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.083461 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:49:49.084130 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.084113 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-155.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:49:49.084234 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.084130 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-155.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:49:49.084234 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.084147 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-155.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:49:49.084234 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.084154 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-155.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:49:49.084234 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.084161 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-155.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:49:49.084234 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.084164 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-155.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:49:49.087077 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.087060 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-155.ec2.internal" Jul 17 17:49:49.087156 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.087093 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 17:49:49.087881 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.087862 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-155.ec2.internal" event="NodeHasSufficientMemory" Jul 17 17:49:49.087966 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.087895 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-155.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 17:49:49.087966 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.087910 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-155.ec2.internal" event="NodeHasSufficientPID" Jul 17 17:49:49.106997 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:49.106980 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-155.ec2.internal\" not found" node="ip-10-0-142-155.ec2.internal" Jul 17 17:49:49.111244 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:49.111226 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-155.ec2.internal\" not found" node="ip-10-0-142-155.ec2.internal" Jul 17 17:49:49.141490 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.141467 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7aea52351981b999e804455ac7eec0e6-config\") pod \"kube-apiserver-proxy-ip-10-0-142-155.ec2.internal\" (UID: \"7aea52351981b999e804455ac7eec0e6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-155.ec2.internal" Jul 17 17:49:49.141621 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.141496 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f271b5630fe33a388a47016c46496a70-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-155.ec2.internal\" (UID: \"f271b5630fe33a388a47016c46496a70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-155.ec2.internal" Jul 17 17:49:49.141621 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.141515 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f271b5630fe33a388a47016c46496a70-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-155.ec2.internal\" (UID: \"f271b5630fe33a388a47016c46496a70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-155.ec2.internal" Jul 17 17:49:49.143576 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:49.143558 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-155.ec2.internal\" not found" Jul 17 17:49:49.242364 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.242328 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7aea52351981b999e804455ac7eec0e6-config\") pod \"kube-apiserver-proxy-ip-10-0-142-155.ec2.internal\" (UID: \"7aea52351981b999e804455ac7eec0e6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-155.ec2.internal" Jul 17 17:49:49.242472 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.242378 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f271b5630fe33a388a47016c46496a70-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-155.ec2.internal\" (UID: \"f271b5630fe33a388a47016c46496a70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-155.ec2.internal" Jul 17 17:49:49.242472 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.242397 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f271b5630fe33a388a47016c46496a70-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-155.ec2.internal\" (UID: \"f271b5630fe33a388a47016c46496a70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-155.ec2.internal" Jul 17 17:49:49.242472 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.242435 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f271b5630fe33a388a47016c46496a70-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-155.ec2.internal\" (UID: \"f271b5630fe33a388a47016c46496a70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-155.ec2.internal" Jul 17 17:49:49.242472 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.242449 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7aea52351981b999e804455ac7eec0e6-config\") pod \"kube-apiserver-proxy-ip-10-0-142-155.ec2.internal\" (UID: \"7aea52351981b999e804455ac7eec0e6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-155.ec2.internal" Jul 17 17:49:49.242594 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.242457 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f271b5630fe33a388a47016c46496a70-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-155.ec2.internal\" (UID: \"f271b5630fe33a388a47016c46496a70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-155.ec2.internal" Jul 17 17:49:49.244441 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:49.244427 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-155.ec2.internal\" not found" Jul 17 17:49:49.345380 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:49.345285 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-155.ec2.internal\" not found" Jul 17 17:49:49.409516 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.409484 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-155.ec2.internal" Jul 17 17:49:49.415397 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.415375 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-155.ec2.internal" Jul 17 17:49:49.445674 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:49.445642 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-155.ec2.internal\" not found" Jul 17 17:49:49.546279 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:49.546240 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-155.ec2.internal\" not found" Jul 17 17:49:49.646831 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:49.646745 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-155.ec2.internal\" not found" Jul 17 17:49:49.747392 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:49.747337 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-155.ec2.internal\" not found" Jul 17 17:49:49.762527 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.762502 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 17 17:49:49.762677 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.762655 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 17:49:49.762715 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.762664 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 17:49:49.819455 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.819428 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 17:49:49.837831 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.837794 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 17:44:48 +0000 UTC" deadline="2028-02-28 04:16:03.464245305 +0000 UTC" Jul 17 17:49:49.837831 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:49.837825 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14170h26m13.62642332s" Jul 17 17:49:49.848458 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:49.848431 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-155.ec2.internal\" not found" Jul 17 17:49:49.949207 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:49.949134 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-155.ec2.internal\" not found" Jul 17 17:49:50.003618 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:50.003580 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf271b5630fe33a388a47016c46496a70.slice/crio-038edac107abdecb8fd685fd73a15ae88d3df905076f3b06ca4005e9388b3bd4 WatchSource:0}: Error finding container 038edac107abdecb8fd685fd73a15ae88d3df905076f3b06ca4005e9388b3bd4: Status 404 returned error can't find the container with id 038edac107abdecb8fd685fd73a15ae88d3df905076f3b06ca4005e9388b3bd4 Jul 17 17:49:50.004166 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:50.004130 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aea52351981b999e804455ac7eec0e6.slice/crio-311b35194c1b6d78fc45c378c0e30d96c96c634162197efa70f417c9f0620403 WatchSource:0}: Error finding container 311b35194c1b6d78fc45c378c0e30d96c96c634162197efa70f417c9f0620403: Status 404 returned error can't find the container with id 311b35194c1b6d78fc45c378c0e30d96c96c634162197efa70f417c9f0620403 Jul 17 17:49:50.009291 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.009279 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 17:49:50.049991 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:50.049950 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-155.ec2.internal\" not found" Jul 17 17:49:50.150535 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:50.150510 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-155.ec2.internal\" not found" Jul 17 17:49:50.250946 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:50.250917 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-155.ec2.internal\" not found" Jul 17 17:49:50.351614 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:50.351578 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-155.ec2.internal\" not found" Jul 17 17:49:50.420952 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.420921 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 17:49:50.438785 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.438760 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-155.ec2.internal" Jul 17 17:49:50.446963 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.446942 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 17:49:50.447715 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.447698 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-155.ec2.internal" Jul 17 17:49:50.463928 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.463899 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 17:49:50.741746 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.741585 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 17:49:50.813788 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.813748 2574 apiserver.go:52] "Watching apiserver" Jul 17 17:49:50.821537 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.821509 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 17 17:49:50.826597 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.826569 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-db2n2","kube-system/kube-apiserver-proxy-ip-10-0-142-155.ec2.internal","openshift-cluster-node-tuning-operator/tuned-7vcqg","openshift-image-registry/node-ca-sbkl7","openshift-multus/multus-hzh2t","openshift-network-diagnostics/network-check-target-mp8l2","openshift-network-operator/iptables-alerter-877hk","openshift-ovn-kubernetes/ovnkube-node-ddl85","kube-system/global-pull-secret-syncer-4xjqt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6","openshift-dns/node-resolver-sfcxh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-155.ec2.internal","openshift-multus/multus-additional-cni-plugins-ncm4s","openshift-multus/network-metrics-daemon-r8hm5"] Jul 17 17:49:50.829461 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.829434 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-877hk" Jul 17 17:49:50.831993 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.831975 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 17 17:49:50.831993 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.831988 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dbjcz\"" Jul 17 17:49:50.832151 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.832007 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.832151 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.832024 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 17 17:49:50.832411 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.832394 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 17 17:49:50.835062 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.835044 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 17 17:49:50.835292 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.835265 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mlfbk\"" Jul 17 17:49:50.835378 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.835333 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 17 17:49:50.835478 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.835464 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 17 17:49:50.835569 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.835552 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.836052 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.836031 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 17 17:49:50.836169 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.836152 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sbkl7" Jul 17 17:49:50.836221 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.836033 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 17 17:49:50.837942 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.837889 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 17 17:49:50.838027 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.837967 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gqkb2\"" Jul 17 17:49:50.838104 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.838082 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 17 17:49:50.838211 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.838188 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 17 17:49:50.838718 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.838704 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.838904 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.838885 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 17 17:49:50.839166 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.839148 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 17 17:49:50.839166 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.839163 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kpf9p\"" Jul 17 17:49:50.839284 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.839181 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 17 17:49:50.841613 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.840794 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:49:50.841613 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:50.840863 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8l2" podUID="6f28f659-4138-4e06-8674-b6e6f8ed3172" Jul 17 17:49:50.841613 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.841182 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 17 17:49:50.841613 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.841440 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8gjdw\"" Jul 17 17:49:50.842818 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.842771 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:49:50.842903 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:50.842842 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8hm5" podUID="315910cc-963f-4190-98a8-4453798d7fc7" Jul 17 17:49:50.844992 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.844974 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-db2n2" Jul 17 17:49:50.847580 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.847561 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:50.848192 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.848172 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 17 17:49:50.848288 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.848231 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-h5hft\"" Jul 17 17:49:50.848367 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.848318 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 17 17:49:50.849748 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.849726 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 17 17:49:50.850106 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.850091 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bpsq4\"" Jul 17 17:49:50.850220 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.850204 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 17 17:49:50.850421 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.850409 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:50.850492 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.850432 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 17 17:49:50.850716 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.850702 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f044d914-e059-4c54-9ab3-b933c612b30a-cnibin\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.850776 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.850723 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nx4h\" (UniqueName: \"kubernetes.io/projected/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-kube-api-access-8nx4h\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.850776 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.850764 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-host-run-netns\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.850865 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.850782 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-host-var-lib-cni-multus\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.850865 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.850798 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-etc-kubernetes\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.850865 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.850835 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f044d914-e059-4c54-9ab3-b933c612b30a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.850987 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.850865 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-etc-modprobe-d\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.850987 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.850888 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-etc-kubernetes\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.850987 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.850911 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-etc-systemd\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.850987 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.850935 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8015312-d14e-4a6b-a54d-8a73e8582aa5-host-slash\") pod \"iptables-alerter-877hk\" (UID: \"e8015312-d14e-4a6b-a54d-8a73e8582aa5\") " pod="openshift-network-operator/iptables-alerter-877hk" Jul 17 17:49:50.850987 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.850958 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f044d914-e059-4c54-9ab3-b933c612b30a-cni-binary-copy\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.851238 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.850980 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-lib-modules\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.851238 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851032 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-var-lib-kubelet\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.851238 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851063 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-host\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.851238 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851087 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd6b07f4-0170-4df9-bcca-1e0613fe7079-host\") pod \"node-ca-sbkl7\" (UID: \"cd6b07f4-0170-4df9-bcca-1e0613fe7079\") " pod="openshift-image-registry/node-ca-sbkl7" Jul 17 17:49:50.851238 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851110 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f044d914-e059-4c54-9ab3-b933c612b30a-system-cni-dir\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.851238 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851132 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f044d914-e059-4c54-9ab3-b933c612b30a-os-release\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.851238 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851153 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-system-cni-dir\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.851238 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851184 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-host-run-k8s-cni-cncf-io\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.851238 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851209 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-hostroot\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.851711 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851251 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f044d914-e059-4c54-9ab3-b933c612b30a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.851711 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851295 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-etc-tuned\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.851711 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851328 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-multus-cni-dir\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.851711 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851388 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c0ed2480-e1bb-4002-bb74-b79be477ba8a-multus-daemon-config\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.851711 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851435 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmbs7\" (UniqueName: \"kubernetes.io/projected/c0ed2480-e1bb-4002-bb74-b79be477ba8a-kube-api-access-vmbs7\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.851711 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851470 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-tmp\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.851711 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851501 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f044d914-e059-4c54-9ab3-b933c612b30a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.851711 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851535 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-etc-sysctl-conf\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.851711 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851545 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 17 17:49:50.851711 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851559 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-sys\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.851711 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851581 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-os-release\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.851711 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851604 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-multus-socket-dir-parent\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.851711 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851633 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttt65\" (UniqueName: \"kubernetes.io/projected/f044d914-e059-4c54-9ab3-b933c612b30a-kube-api-access-ttt65\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.851711 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851640 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 17 17:49:50.851711 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851675 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-etc-sysconfig\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.851711 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851696 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-etc-sysctl-d\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.851711 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851717 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjfqg\" (UniqueName: \"kubernetes.io/projected/cd6b07f4-0170-4df9-bcca-1e0613fe7079-kube-api-access-xjfqg\") pod \"node-ca-sbkl7\" (UID: \"cd6b07f4-0170-4df9-bcca-1e0613fe7079\") " pod="openshift-image-registry/node-ca-sbkl7" Jul 17 17:49:50.851711 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851744 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-host-var-lib-cni-bin\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.852538 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851764 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-host-var-lib-kubelet\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.852538 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.851784 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-multus-conf-dir\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.852538 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.852081 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 17 17:49:50.852538 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.852117 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sfcxh" Jul 17 17:49:50.852538 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.852135 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6tnn\" (UniqueName: \"kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn\") pod \"network-check-target-mp8l2\" (UID: \"6f28f659-4138-4e06-8674-b6e6f8ed3172\") " pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:49:50.852538 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.852258 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-run\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.852538 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.852386 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cd6b07f4-0170-4df9-bcca-1e0613fe7079-serviceca\") pod \"node-ca-sbkl7\" (UID: \"cd6b07f4-0170-4df9-bcca-1e0613fe7079\") " pod="openshift-image-registry/node-ca-sbkl7" Jul 17 17:49:50.852538 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.852472 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-cnibin\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.852538 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.852488 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0ed2480-e1bb-4002-bb74-b79be477ba8a-cni-binary-copy\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.852899 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.852577 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-host-run-multus-certs\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.852899 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.852610 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e8015312-d14e-4a6b-a54d-8a73e8582aa5-iptables-alerter-script\") pod \"iptables-alerter-877hk\" (UID: \"e8015312-d14e-4a6b-a54d-8a73e8582aa5\") " pod="openshift-network-operator/iptables-alerter-877hk" Jul 17 17:49:50.852899 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.852625 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhzz8\" (UniqueName: \"kubernetes.io/projected/e8015312-d14e-4a6b-a54d-8a73e8582aa5-kube-api-access-jhzz8\") pod \"iptables-alerter-877hk\" (UID: \"e8015312-d14e-4a6b-a54d-8a73e8582aa5\") " pod="openshift-network-operator/iptables-alerter-877hk" Jul 17 17:49:50.852899 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.852710 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 17 17:49:50.852899 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.852750 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 17 17:49:50.852899 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.852758 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 17 17:49:50.858223 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.855984 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m7vzb\"" Jul 17 17:49:50.858223 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.856233 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 17:49:50.858223 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.856330 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 17 17:49:50.858223 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.856304 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 17 17:49:50.858223 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.856549 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bqlnf\"" Jul 17 17:49:50.858223 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.857417 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:49:50.858223 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:50.857498 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjqt" podUID="fa00821b-8e34-4030-9988-e37c6a12230d" Jul 17 17:49:50.875074 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.875053 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4jnb6" Jul 17 17:49:50.881882 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.881838 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4jnb6" Jul 17 17:49:50.940414 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.940388 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 17 17:49:50.953669 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.953634 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-os-release\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.953817 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.953697 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-host-cni-bin\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:50.953817 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.953726 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-host-cni-netd\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:50.953817 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.953745 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-os-release\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.953817 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.953753 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/55537b77-ff5d-44be-b7f7-8001ab4dd01d-ovnkube-script-lib\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:50.953817 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.953781 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abf9fe12-d673-42ab-9178-66dd5a309513-device-dir\") pod \"aws-ebs-csi-driver-node-wqjx6\" (UID: \"abf9fe12-d673-42ab-9178-66dd5a309513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:50.953979 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.953825 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/db8a7cf2-a598-4192-b51b-815e6ad11718-hosts-file\") pod \"node-resolver-sfcxh\" (UID: \"db8a7cf2-a598-4192-b51b-815e6ad11718\") " pod="openshift-dns/node-resolver-sfcxh" Jul 17 17:49:50.953979 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.953854 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttt65\" (UniqueName: \"kubernetes.io/projected/f044d914-e059-4c54-9ab3-b933c612b30a-kube-api-access-ttt65\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.953979 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.953878 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-etc-sysconfig\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.953979 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.953901 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-etc-sysctl-d\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.953979 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.953925 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-host-var-lib-cni-bin\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.953979 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.953951 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-host-var-lib-kubelet\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.953979 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.953950 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-etc-sysconfig\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.953979 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.953974 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6tnn\" (UniqueName: \"kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn\") pod \"network-check-target-mp8l2\" (UID: \"6f28f659-4138-4e06-8674-b6e6f8ed3172\") " pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:49:50.954476 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954000 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-host-var-lib-cni-bin\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.954476 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954036 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-host-var-lib-kubelet\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.954476 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954034 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e8015312-d14e-4a6b-a54d-8a73e8582aa5-iptables-alerter-script\") pod \"iptables-alerter-877hk\" (UID: \"e8015312-d14e-4a6b-a54d-8a73e8582aa5\") " pod="openshift-network-operator/iptables-alerter-877hk" Jul 17 17:49:50.954476 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954096 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-etc-sysctl-d\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.954476 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954096 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhzz8\" (UniqueName: \"kubernetes.io/projected/e8015312-d14e-4a6b-a54d-8a73e8582aa5-kube-api-access-jhzz8\") pod \"iptables-alerter-877hk\" (UID: \"e8015312-d14e-4a6b-a54d-8a73e8582aa5\") " pod="openshift-network-operator/iptables-alerter-877hk" Jul 17 17:49:50.954476 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954191 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-run\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.954476 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954214 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-cnibin\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.954476 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954241 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0ed2480-e1bb-4002-bb74-b79be477ba8a-cni-binary-copy\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.954476 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954247 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-run\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.954476 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954258 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-cnibin\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.954476 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954267 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-host-run-multus-certs\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.954476 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954287 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-run-openvswitch\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:50.954476 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954322 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-node-log\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:50.954476 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954339 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-host-run-multus-certs\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.954476 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954348 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abf9fe12-d673-42ab-9178-66dd5a309513-socket-dir\") pod \"aws-ebs-csi-driver-node-wqjx6\" (UID: \"abf9fe12-d673-42ab-9178-66dd5a309513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:50.954476 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954402 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret\") pod \"global-pull-secret-syncer-4xjqt\" (UID: \"fa00821b-8e34-4030-9988-e37c6a12230d\") " pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:49:50.954476 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954448 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nx4h\" (UniqueName: \"kubernetes.io/projected/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-kube-api-access-8nx4h\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.955121 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954490 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-host-var-lib-cni-multus\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.955121 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954515 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqbtf\" (UniqueName: \"kubernetes.io/projected/55537b77-ff5d-44be-b7f7-8001ab4dd01d-kube-api-access-rqbtf\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:50.955121 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954545 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa00821b-8e34-4030-9988-e37c6a12230d-kubelet-config\") pod \"global-pull-secret-syncer-4xjqt\" (UID: \"fa00821b-8e34-4030-9988-e37c6a12230d\") " pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:49:50.955121 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954567 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-host-var-lib-cni-multus\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.955121 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954577 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-etc-modprobe-d\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.955121 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954611 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-etc-kubernetes\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.955121 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954635 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8015312-d14e-4a6b-a54d-8a73e8582aa5-host-slash\") pod \"iptables-alerter-877hk\" (UID: \"e8015312-d14e-4a6b-a54d-8a73e8582aa5\") " pod="openshift-network-operator/iptables-alerter-877hk" Jul 17 17:49:50.955121 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954659 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-host-kubelet\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:50.955121 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954680 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3a6a8a84-5e48-4b57-b683-a29619e9120f-konnectivity-ca\") pod \"konnectivity-agent-db2n2\" (UID: \"3a6a8a84-5e48-4b57-b683-a29619e9120f\") " pod="kube-system/konnectivity-agent-db2n2" Jul 17 17:49:50.955121 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954686 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-etc-modprobe-d\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.955121 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954693 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e8015312-d14e-4a6b-a54d-8a73e8582aa5-iptables-alerter-script\") pod \"iptables-alerter-877hk\" (UID: \"e8015312-d14e-4a6b-a54d-8a73e8582aa5\") " pod="openshift-network-operator/iptables-alerter-877hk" Jul 17 17:49:50.955121 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954726 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-var-lib-kubelet\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.955121 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954730 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8015312-d14e-4a6b-a54d-8a73e8582aa5-host-slash\") pod \"iptables-alerter-877hk\" (UID: \"e8015312-d14e-4a6b-a54d-8a73e8582aa5\") " pod="openshift-network-operator/iptables-alerter-877hk" Jul 17 17:49:50.955121 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954758 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0ed2480-e1bb-4002-bb74-b79be477ba8a-cni-binary-copy\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.955121 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954752 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd6b07f4-0170-4df9-bcca-1e0613fe7079-host\") pod \"node-ca-sbkl7\" (UID: \"cd6b07f4-0170-4df9-bcca-1e0613fe7079\") " pod="openshift-image-registry/node-ca-sbkl7" Jul 17 17:49:50.955121 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954715 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-etc-kubernetes\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.955121 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954775 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-var-lib-kubelet\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.955723 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954803 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd6b07f4-0170-4df9-bcca-1e0613fe7079-host\") pod \"node-ca-sbkl7\" (UID: \"cd6b07f4-0170-4df9-bcca-1e0613fe7079\") " pod="openshift-image-registry/node-ca-sbkl7" Jul 17 17:49:50.955723 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954834 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-host-slash\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:50.955723 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954849 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-etc-openvswitch\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:50.955723 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954863 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:50.955723 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954902 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/55537b77-ff5d-44be-b7f7-8001ab4dd01d-ovnkube-config\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:50.955723 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954916 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8qvr\" (UniqueName: \"kubernetes.io/projected/abf9fe12-d673-42ab-9178-66dd5a309513-kube-api-access-c8qvr\") pod \"aws-ebs-csi-driver-node-wqjx6\" (UID: \"abf9fe12-d673-42ab-9178-66dd5a309513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:50.955723 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954938 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f044d914-e059-4c54-9ab3-b933c612b30a-system-cni-dir\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.955723 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954965 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f044d914-e059-4c54-9ab3-b933c612b30a-os-release\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.955723 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.954984 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-system-cni-dir\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.955723 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955000 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-hostroot\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.955723 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955041 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abf9fe12-d673-42ab-9178-66dd5a309513-sys-fs\") pod \"aws-ebs-csi-driver-node-wqjx6\" (UID: \"abf9fe12-d673-42ab-9178-66dd5a309513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:50.955723 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955051 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f044d914-e059-4c54-9ab3-b933c612b30a-os-release\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.955723 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955051 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-system-cni-dir\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.955723 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955053 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f044d914-e059-4c54-9ab3-b933c612b30a-system-cni-dir\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.955723 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955055 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs\") pod \"network-metrics-daemon-r8hm5\" (UID: \"315910cc-963f-4190-98a8-4453798d7fc7\") " pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:49:50.955723 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955096 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-hostroot\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.955723 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955142 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tngrt\" (UniqueName: \"kubernetes.io/projected/315910cc-963f-4190-98a8-4453798d7fc7-kube-api-access-tngrt\") pod \"network-metrics-daemon-r8hm5\" (UID: \"315910cc-963f-4190-98a8-4453798d7fc7\") " pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:49:50.956465 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955184 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f044d914-e059-4c54-9ab3-b933c612b30a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.956465 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955207 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmbs7\" (UniqueName: \"kubernetes.io/projected/c0ed2480-e1bb-4002-bb74-b79be477ba8a-kube-api-access-vmbs7\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.956465 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955229 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-host-run-netns\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:50.956465 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955255 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8a7cf2-a598-4192-b51b-815e6ad11718-tmp-dir\") pod \"node-resolver-sfcxh\" (UID: \"db8a7cf2-a598-4192-b51b-815e6ad11718\") " pod="openshift-dns/node-resolver-sfcxh" Jul 17 17:49:50.956465 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955277 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3a6a8a84-5e48-4b57-b683-a29619e9120f-agent-certs\") pod \"konnectivity-agent-db2n2\" (UID: \"3a6a8a84-5e48-4b57-b683-a29619e9120f\") " pod="kube-system/konnectivity-agent-db2n2" Jul 17 17:49:50.956465 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955380 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f044d914-e059-4c54-9ab3-b933c612b30a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.956465 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955408 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-etc-sysctl-conf\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.956465 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955430 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-multus-socket-dir-parent\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.956465 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955520 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/55537b77-ff5d-44be-b7f7-8001ab4dd01d-ovn-node-metrics-cert\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:50.956465 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955527 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-multus-socket-dir-parent\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.956465 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955557 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abf9fe12-d673-42ab-9178-66dd5a309513-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wqjx6\" (UID: \"abf9fe12-d673-42ab-9178-66dd5a309513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:50.956465 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955561 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-etc-sysctl-conf\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.956465 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955602 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abf9fe12-d673-42ab-9178-66dd5a309513-registration-dir\") pod \"aws-ebs-csi-driver-node-wqjx6\" (UID: \"abf9fe12-d673-42ab-9178-66dd5a309513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:50.956465 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955621 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjfqg\" (UniqueName: \"kubernetes.io/projected/cd6b07f4-0170-4df9-bcca-1e0613fe7079-kube-api-access-xjfqg\") pod \"node-ca-sbkl7\" (UID: \"cd6b07f4-0170-4df9-bcca-1e0613fe7079\") " pod="openshift-image-registry/node-ca-sbkl7" Jul 17 17:49:50.956465 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955647 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-multus-conf-dir\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.956465 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955665 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-host-run-ovn-kubernetes\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:50.956465 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955665 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f044d914-e059-4c54-9ab3-b933c612b30a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.957032 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955816 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cd6b07f4-0170-4df9-bcca-1e0613fe7079-serviceca\") pod \"node-ca-sbkl7\" (UID: \"cd6b07f4-0170-4df9-bcca-1e0613fe7079\") " pod="openshift-image-registry/node-ca-sbkl7" Jul 17 17:49:50.957032 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955842 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-run-systemd\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:50.957032 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955866 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f044d914-e059-4c54-9ab3-b933c612b30a-cnibin\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.957032 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955877 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-multus-conf-dir\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.957032 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955888 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-host-run-netns\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.957032 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955897 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f044d914-e059-4c54-9ab3-b933c612b30a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.957032 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955922 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-host-run-netns\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.957032 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955926 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f044d914-e059-4c54-9ab3-b933c612b30a-cnibin\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.957032 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955964 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-etc-kubernetes\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.957032 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.955985 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-run-ovn\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:50.957032 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956000 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa00821b-8e34-4030-9988-e37c6a12230d-dbus\") pod \"global-pull-secret-syncer-4xjqt\" (UID: \"fa00821b-8e34-4030-9988-e37c6a12230d\") " pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:49:50.957032 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956020 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f044d914-e059-4c54-9ab3-b933c612b30a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.957032 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956023 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-etc-kubernetes\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.957032 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956038 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-etc-systemd\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.957032 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956055 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-systemd-units\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:50.957032 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956069 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-var-lib-openvswitch\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:50.957032 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956078 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-etc-systemd\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.957520 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956085 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f044d914-e059-4c54-9ab3-b933c612b30a-cni-binary-copy\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.957520 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956113 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-lib-modules\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.957520 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956148 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f044d914-e059-4c54-9ab3-b933c612b30a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.957520 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956188 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-host\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.957520 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956194 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cd6b07f4-0170-4df9-bcca-1e0613fe7079-serviceca\") pod \"node-ca-sbkl7\" (UID: \"cd6b07f4-0170-4df9-bcca-1e0613fe7079\") " pod="openshift-image-registry/node-ca-sbkl7" Jul 17 17:49:50.957520 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956239 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abf9fe12-d673-42ab-9178-66dd5a309513-etc-selinux\") pod \"aws-ebs-csi-driver-node-wqjx6\" (UID: \"abf9fe12-d673-42ab-9178-66dd5a309513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:50.957520 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956257 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-host\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.957520 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956267 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t67sx\" (UniqueName: \"kubernetes.io/projected/db8a7cf2-a598-4192-b51b-815e6ad11718-kube-api-access-t67sx\") pod \"node-resolver-sfcxh\" (UID: \"db8a7cf2-a598-4192-b51b-815e6ad11718\") " pod="openshift-dns/node-resolver-sfcxh" Jul 17 17:49:50.957520 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956263 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-lib-modules\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.957520 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956298 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-host-run-k8s-cni-cncf-io\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.957520 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956331 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-etc-tuned\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.957520 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956375 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-multus-cni-dir\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.957520 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956385 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-host-run-k8s-cni-cncf-io\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.957520 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956400 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c0ed2480-e1bb-4002-bb74-b79be477ba8a-multus-daemon-config\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.957520 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956422 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-log-socket\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:50.957520 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956444 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/55537b77-ff5d-44be-b7f7-8001ab4dd01d-env-overrides\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:50.957520 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956478 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-tmp\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.958017 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956497 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-sys\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.958017 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956543 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f044d914-e059-4c54-9ab3-b933c612b30a-cni-binary-copy\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.958017 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956543 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0ed2480-e1bb-4002-bb74-b79be477ba8a-multus-cni-dir\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.958017 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956581 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-sys\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.958017 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956630 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 17 17:49:50.958017 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.956824 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c0ed2480-e1bb-4002-bb74-b79be477ba8a-multus-daemon-config\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.959252 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.959234 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-etc-tuned\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.959378 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.959334 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-tmp\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.964385 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:50.964315 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:49:50.964385 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:50.964374 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:49:50.964385 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:50.964388 2574 projected.go:196] Error preparing data for projected volume kube-api-access-t6tnn for pod openshift-network-diagnostics/network-check-target-mp8l2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:49:50.966528 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:50.964699 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn podName:6f28f659-4138-4e06-8674-b6e6f8ed3172 nodeName:}" failed. No retries permitted until 2026-07-17 17:49:51.464467164 +0000 UTC m=+3.052419880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t6tnn" (UniqueName: "kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn") pod "network-check-target-mp8l2" (UID: "6f28f659-4138-4e06-8674-b6e6f8ed3172") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:49:50.966528 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.965624 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttt65\" (UniqueName: \"kubernetes.io/projected/f044d914-e059-4c54-9ab3-b933c612b30a-kube-api-access-ttt65\") pod \"multus-additional-cni-plugins-ncm4s\" (UID: \"f044d914-e059-4c54-9ab3-b933c612b30a\") " pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:50.966528 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.966192 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmbs7\" (UniqueName: \"kubernetes.io/projected/c0ed2480-e1bb-4002-bb74-b79be477ba8a-kube-api-access-vmbs7\") pod \"multus-hzh2t\" (UID: \"c0ed2480-e1bb-4002-bb74-b79be477ba8a\") " pod="openshift-multus/multus-hzh2t" Jul 17 17:49:50.966528 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.966501 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nx4h\" (UniqueName: \"kubernetes.io/projected/bcfe772e-cd4d-4f4b-92b3-49d68f4e4176-kube-api-access-8nx4h\") pod \"tuned-7vcqg\" (UID: \"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176\") " pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:50.966737 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.966718 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjfqg\" (UniqueName: \"kubernetes.io/projected/cd6b07f4-0170-4df9-bcca-1e0613fe7079-kube-api-access-xjfqg\") pod \"node-ca-sbkl7\" (UID: \"cd6b07f4-0170-4df9-bcca-1e0613fe7079\") " pod="openshift-image-registry/node-ca-sbkl7" Jul 17 17:49:50.966898 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.966873 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhzz8\" (UniqueName: \"kubernetes.io/projected/e8015312-d14e-4a6b-a54d-8a73e8582aa5-kube-api-access-jhzz8\") pod \"iptables-alerter-877hk\" (UID: \"e8015312-d14e-4a6b-a54d-8a73e8582aa5\") " pod="openshift-network-operator/iptables-alerter-877hk" Jul 17 17:49:50.984566 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.984505 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-155.ec2.internal" event={"ID":"f271b5630fe33a388a47016c46496a70","Type":"ContainerStarted","Data":"038edac107abdecb8fd685fd73a15ae88d3df905076f3b06ca4005e9388b3bd4"} Jul 17 17:49:50.985603 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:50.985578 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-155.ec2.internal" event={"ID":"7aea52351981b999e804455ac7eec0e6","Type":"ContainerStarted","Data":"311b35194c1b6d78fc45c378c0e30d96c96c634162197efa70f417c9f0620403"} Jul 17 17:49:51.057071 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.056975 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/55537b77-ff5d-44be-b7f7-8001ab4dd01d-ovn-node-metrics-cert\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.057071 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057022 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abf9fe12-d673-42ab-9178-66dd5a309513-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wqjx6\" (UID: \"abf9fe12-d673-42ab-9178-66dd5a309513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:51.057071 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057044 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abf9fe12-d673-42ab-9178-66dd5a309513-registration-dir\") pod \"aws-ebs-csi-driver-node-wqjx6\" (UID: \"abf9fe12-d673-42ab-9178-66dd5a309513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:51.057294 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057109 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/abf9fe12-d673-42ab-9178-66dd5a309513-registration-dir\") pod \"aws-ebs-csi-driver-node-wqjx6\" (UID: \"abf9fe12-d673-42ab-9178-66dd5a309513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:51.057294 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057114 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/abf9fe12-d673-42ab-9178-66dd5a309513-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wqjx6\" (UID: \"abf9fe12-d673-42ab-9178-66dd5a309513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:51.057294 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057146 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-host-run-ovn-kubernetes\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.057294 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057169 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-run-systemd\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.057294 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057195 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-run-ovn\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.057294 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057218 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa00821b-8e34-4030-9988-e37c6a12230d-dbus\") pod \"global-pull-secret-syncer-4xjqt\" (UID: \"fa00821b-8e34-4030-9988-e37c6a12230d\") " pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:49:51.057294 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057222 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-run-systemd\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.057294 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057260 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-run-ovn\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.057294 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057184 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-host-run-ovn-kubernetes\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.057294 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057284 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-systemd-units\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.057675 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057314 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-var-lib-openvswitch\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.057675 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057333 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-systemd-units\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.057675 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057347 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa00821b-8e34-4030-9988-e37c6a12230d-dbus\") pod \"global-pull-secret-syncer-4xjqt\" (UID: \"fa00821b-8e34-4030-9988-e37c6a12230d\") " pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:49:51.057675 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057340 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abf9fe12-d673-42ab-9178-66dd5a309513-etc-selinux\") pod \"aws-ebs-csi-driver-node-wqjx6\" (UID: \"abf9fe12-d673-42ab-9178-66dd5a309513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:51.057675 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057395 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-var-lib-openvswitch\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.057675 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057402 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t67sx\" (UniqueName: \"kubernetes.io/projected/db8a7cf2-a598-4192-b51b-815e6ad11718-kube-api-access-t67sx\") pod \"node-resolver-sfcxh\" (UID: \"db8a7cf2-a598-4192-b51b-815e6ad11718\") " pod="openshift-dns/node-resolver-sfcxh" Jul 17 17:49:51.057675 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057445 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-log-socket\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.057675 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057468 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/55537b77-ff5d-44be-b7f7-8001ab4dd01d-env-overrides\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.057675 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057477 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-log-socket\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.057675 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057447 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/abf9fe12-d673-42ab-9178-66dd5a309513-etc-selinux\") pod \"aws-ebs-csi-driver-node-wqjx6\" (UID: \"abf9fe12-d673-42ab-9178-66dd5a309513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:51.057675 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057492 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-host-cni-bin\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.057675 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057511 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-host-cni-netd\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.057675 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057530 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/55537b77-ff5d-44be-b7f7-8001ab4dd01d-ovnkube-script-lib\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.057675 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057550 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abf9fe12-d673-42ab-9178-66dd5a309513-device-dir\") pod \"aws-ebs-csi-driver-node-wqjx6\" (UID: \"abf9fe12-d673-42ab-9178-66dd5a309513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:51.057675 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057569 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/db8a7cf2-a598-4192-b51b-815e6ad11718-hosts-file\") pod \"node-resolver-sfcxh\" (UID: \"db8a7cf2-a598-4192-b51b-815e6ad11718\") " pod="openshift-dns/node-resolver-sfcxh" Jul 17 17:49:51.057675 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057607 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-run-openvswitch\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.057675 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057625 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-node-log\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.058202 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057642 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abf9fe12-d673-42ab-9178-66dd5a309513-socket-dir\") pod \"aws-ebs-csi-driver-node-wqjx6\" (UID: \"abf9fe12-d673-42ab-9178-66dd5a309513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:51.058202 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057663 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret\") pod \"global-pull-secret-syncer-4xjqt\" (UID: \"fa00821b-8e34-4030-9988-e37c6a12230d\") " pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:49:51.058202 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057689 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqbtf\" (UniqueName: \"kubernetes.io/projected/55537b77-ff5d-44be-b7f7-8001ab4dd01d-kube-api-access-rqbtf\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.058202 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057699 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/abf9fe12-d673-42ab-9178-66dd5a309513-device-dir\") pod \"aws-ebs-csi-driver-node-wqjx6\" (UID: \"abf9fe12-d673-42ab-9178-66dd5a309513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:51.058202 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057710 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa00821b-8e34-4030-9988-e37c6a12230d-kubelet-config\") pod \"global-pull-secret-syncer-4xjqt\" (UID: \"fa00821b-8e34-4030-9988-e37c6a12230d\") " pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:49:51.058202 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057738 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-host-kubelet\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.058202 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057742 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-host-cni-bin\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.058202 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057760 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3a6a8a84-5e48-4b57-b683-a29619e9120f-konnectivity-ca\") pod \"konnectivity-agent-db2n2\" (UID: \"3a6a8a84-5e48-4b57-b683-a29619e9120f\") " pod="kube-system/konnectivity-agent-db2n2" Jul 17 17:49:51.058202 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057780 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-host-cni-netd\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.058202 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057785 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-host-slash\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.058202 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-etc-openvswitch\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.058202 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057834 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-run-openvswitch\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.058202 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057833 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.058202 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:51.057851 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:49:51.058202 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057861 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/55537b77-ff5d-44be-b7f7-8001ab4dd01d-ovnkube-config\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.058202 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057867 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.058202 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057877 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8qvr\" (UniqueName: \"kubernetes.io/projected/abf9fe12-d673-42ab-9178-66dd5a309513-kube-api-access-c8qvr\") pod \"aws-ebs-csi-driver-node-wqjx6\" (UID: \"abf9fe12-d673-42ab-9178-66dd5a309513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:51.058760 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:51.057920 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret podName:fa00821b-8e34-4030-9988-e37c6a12230d nodeName:}" failed. No retries permitted until 2026-07-17 17:49:51.557903578 +0000 UTC m=+3.145856267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret") pod "global-pull-secret-syncer-4xjqt" (UID: "fa00821b-8e34-4030-9988-e37c6a12230d") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:49:51.058760 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057936 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abf9fe12-d673-42ab-9178-66dd5a309513-sys-fs\") pod \"aws-ebs-csi-driver-node-wqjx6\" (UID: \"abf9fe12-d673-42ab-9178-66dd5a309513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:51.058760 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057960 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs\") pod \"network-metrics-daemon-r8hm5\" (UID: \"315910cc-963f-4190-98a8-4453798d7fc7\") " pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:49:51.058760 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057983 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tngrt\" (UniqueName: \"kubernetes.io/projected/315910cc-963f-4190-98a8-4453798d7fc7-kube-api-access-tngrt\") pod \"network-metrics-daemon-r8hm5\" (UID: \"315910cc-963f-4190-98a8-4453798d7fc7\") " pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:49:51.058760 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.058006 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-host-run-netns\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.058760 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.058027 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8a7cf2-a598-4192-b51b-815e6ad11718-tmp-dir\") pod \"node-resolver-sfcxh\" (UID: \"db8a7cf2-a598-4192-b51b-815e6ad11718\") " pod="openshift-dns/node-resolver-sfcxh" Jul 17 17:49:51.058760 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.058049 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3a6a8a84-5e48-4b57-b683-a29619e9120f-agent-certs\") pod \"konnectivity-agent-db2n2\" (UID: \"3a6a8a84-5e48-4b57-b683-a29619e9120f\") " pod="kube-system/konnectivity-agent-db2n2" Jul 17 17:49:51.058760 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.058052 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa00821b-8e34-4030-9988-e37c6a12230d-kubelet-config\") pod \"global-pull-secret-syncer-4xjqt\" (UID: \"fa00821b-8e34-4030-9988-e37c6a12230d\") " pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:49:51.058760 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.058108 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-etc-openvswitch\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.058760 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.058117 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/db8a7cf2-a598-4192-b51b-815e6ad11718-hosts-file\") pod \"node-resolver-sfcxh\" (UID: \"db8a7cf2-a598-4192-b51b-815e6ad11718\") " pod="openshift-dns/node-resolver-sfcxh" Jul 17 17:49:51.058760 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.058119 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-node-log\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.058760 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.058146 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-host-slash\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.058760 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.057808 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/55537b77-ff5d-44be-b7f7-8001ab4dd01d-env-overrides\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.058760 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.058295 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/55537b77-ff5d-44be-b7f7-8001ab4dd01d-ovnkube-script-lib\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.058760 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:51.058403 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:49:51.058760 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:51.058458 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs podName:315910cc-963f-4190-98a8-4453798d7fc7 nodeName:}" failed. No retries permitted until 2026-07-17 17:49:51.558442844 +0000 UTC m=+3.146395524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs") pod "network-metrics-daemon-r8hm5" (UID: "315910cc-963f-4190-98a8-4453798d7fc7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:49:51.059489 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.058482 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db8a7cf2-a598-4192-b51b-815e6ad11718-tmp-dir\") pod \"node-resolver-sfcxh\" (UID: \"db8a7cf2-a598-4192-b51b-815e6ad11718\") " pod="openshift-dns/node-resolver-sfcxh" Jul 17 17:49:51.059489 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.058495 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/55537b77-ff5d-44be-b7f7-8001ab4dd01d-ovnkube-config\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.059489 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.058503 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/abf9fe12-d673-42ab-9178-66dd5a309513-sys-fs\") pod \"aws-ebs-csi-driver-node-wqjx6\" (UID: \"abf9fe12-d673-42ab-9178-66dd5a309513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:51.059489 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.058551 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/abf9fe12-d673-42ab-9178-66dd5a309513-socket-dir\") pod \"aws-ebs-csi-driver-node-wqjx6\" (UID: \"abf9fe12-d673-42ab-9178-66dd5a309513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:51.059489 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.058630 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-host-run-netns\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.059489 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.058669 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/55537b77-ff5d-44be-b7f7-8001ab4dd01d-host-kubelet\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.059489 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.058970 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3a6a8a84-5e48-4b57-b683-a29619e9120f-konnectivity-ca\") pod \"konnectivity-agent-db2n2\" (UID: \"3a6a8a84-5e48-4b57-b683-a29619e9120f\") " pod="kube-system/konnectivity-agent-db2n2" Jul 17 17:49:51.060665 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.060642 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3a6a8a84-5e48-4b57-b683-a29619e9120f-agent-certs\") pod \"konnectivity-agent-db2n2\" (UID: \"3a6a8a84-5e48-4b57-b683-a29619e9120f\") " pod="kube-system/konnectivity-agent-db2n2" Jul 17 17:49:51.060864 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.060851 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/55537b77-ff5d-44be-b7f7-8001ab4dd01d-ovn-node-metrics-cert\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.067925 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.067901 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8qvr\" (UniqueName: \"kubernetes.io/projected/abf9fe12-d673-42ab-9178-66dd5a309513-kube-api-access-c8qvr\") pod \"aws-ebs-csi-driver-node-wqjx6\" (UID: \"abf9fe12-d673-42ab-9178-66dd5a309513\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:51.068054 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.068038 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t67sx\" (UniqueName: \"kubernetes.io/projected/db8a7cf2-a598-4192-b51b-815e6ad11718-kube-api-access-t67sx\") pod \"node-resolver-sfcxh\" (UID: \"db8a7cf2-a598-4192-b51b-815e6ad11718\") " pod="openshift-dns/node-resolver-sfcxh" Jul 17 17:49:51.068465 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.068443 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqbtf\" (UniqueName: \"kubernetes.io/projected/55537b77-ff5d-44be-b7f7-8001ab4dd01d-kube-api-access-rqbtf\") pod \"ovnkube-node-ddl85\" (UID: \"55537b77-ff5d-44be-b7f7-8001ab4dd01d\") " pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.068900 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.068884 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tngrt\" (UniqueName: \"kubernetes.io/projected/315910cc-963f-4190-98a8-4453798d7fc7-kube-api-access-tngrt\") pod \"network-metrics-daemon-r8hm5\" (UID: \"315910cc-963f-4190-98a8-4453798d7fc7\") " pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:49:51.074719 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.074702 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 17:49:51.143747 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.143716 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-877hk" Jul 17 17:49:51.157195 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.157171 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ncm4s" Jul 17 17:49:51.165780 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.165757 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" Jul 17 17:49:51.171309 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.171285 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sbkl7" Jul 17 17:49:51.178958 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.178938 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hzh2t" Jul 17 17:49:51.187541 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.187515 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-db2n2" Jul 17 17:49:51.194153 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.194131 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:49:51.202727 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.202707 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" Jul 17 17:49:51.207127 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.207110 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sfcxh" Jul 17 17:49:51.561758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.561717 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret\") pod \"global-pull-secret-syncer-4xjqt\" (UID: \"fa00821b-8e34-4030-9988-e37c6a12230d\") " pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:49:51.561937 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.561784 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs\") pod \"network-metrics-daemon-r8hm5\" (UID: \"315910cc-963f-4190-98a8-4453798d7fc7\") " pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:49:51.561937 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.561846 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6tnn\" (UniqueName: \"kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn\") pod \"network-check-target-mp8l2\" (UID: \"6f28f659-4138-4e06-8674-b6e6f8ed3172\") " pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:49:51.561937 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:51.561883 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:49:51.562055 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:51.561938 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:49:51.562055 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:51.561952 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret podName:fa00821b-8e34-4030-9988-e37c6a12230d nodeName:}" failed. No retries permitted until 2026-07-17 17:49:52.561936841 +0000 UTC m=+4.149889507 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret") pod "global-pull-secret-syncer-4xjqt" (UID: "fa00821b-8e34-4030-9988-e37c6a12230d") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:49:51.562055 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:51.561954 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:49:51.562055 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:51.561970 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:49:51.562055 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:51.561983 2574 projected.go:196] Error preparing data for projected volume kube-api-access-t6tnn for pod openshift-network-diagnostics/network-check-target-mp8l2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:49:51.562055 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:51.561987 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs podName:315910cc-963f-4190-98a8-4453798d7fc7 nodeName:}" failed. No retries permitted until 2026-07-17 17:49:52.561973236 +0000 UTC m=+4.149925920 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs") pod "network-metrics-daemon-r8hm5" (UID: "315910cc-963f-4190-98a8-4453798d7fc7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:49:51.562055 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:51.562016 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn podName:6f28f659-4138-4e06-8674-b6e6f8ed3172 nodeName:}" failed. No retries permitted until 2026-07-17 17:49:52.562007972 +0000 UTC m=+4.149960638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t6tnn" (UniqueName: "kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn") pod "network-check-target-mp8l2" (UID: "6f28f659-4138-4e06-8674-b6e6f8ed3172") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:49:51.884854 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.884769 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 17:44:50 +0000 UTC" deadline="2028-02-04 02:04:45.705200184 +0000 UTC" Jul 17 17:49:51.884854 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.884796 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13592h14m53.82040686s" Jul 17 17:49:51.898539 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:51.898508 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8015312_d14e_4a6b_a54d_8a73e8582aa5.slice/crio-d4efbc0f056ac41ebc1c93f7a2d0e43e82be385a8eef73cf031eb1cd477d1438 WatchSource:0}: Error finding container d4efbc0f056ac41ebc1c93f7a2d0e43e82be385a8eef73cf031eb1cd477d1438: Status 404 returned error can't find the container with id d4efbc0f056ac41ebc1c93f7a2d0e43e82be385a8eef73cf031eb1cd477d1438 Jul 17 17:49:51.903240 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:51.903138 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ed2480_e1bb_4002_bb74_b79be477ba8a.slice/crio-d7431f400b0f4e65ff8a40fb217ea09f94a1f3fd57dba9efae418faaea2f81ad WatchSource:0}: Error finding container d7431f400b0f4e65ff8a40fb217ea09f94a1f3fd57dba9efae418faaea2f81ad: Status 404 returned error can't find the container with id d7431f400b0f4e65ff8a40fb217ea09f94a1f3fd57dba9efae418faaea2f81ad Jul 17 17:49:51.904550 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:51.904522 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf9fe12_d673_42ab_9178_66dd5a309513.slice/crio-6fdb81279079a31b66f994c250dbd0c9e3432e909f1a16fe3c08801785b2847e WatchSource:0}: Error finding container 6fdb81279079a31b66f994c250dbd0c9e3432e909f1a16fe3c08801785b2847e: Status 404 returned error can't find the container with id 6fdb81279079a31b66f994c250dbd0c9e3432e909f1a16fe3c08801785b2847e Jul 17 17:49:51.905557 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:51.905126 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf044d914_e059_4c54_9ab3_b933c612b30a.slice/crio-c4fb53986d32022c7556e3326b43ccd192dd33388f8d144028b89d25698d253c WatchSource:0}: Error finding container c4fb53986d32022c7556e3326b43ccd192dd33388f8d144028b89d25698d253c: Status 404 returned error can't find the container with id c4fb53986d32022c7556e3326b43ccd192dd33388f8d144028b89d25698d253c Jul 17 17:49:51.905943 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:51.905923 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55537b77_ff5d_44be_b7f7_8001ab4dd01d.slice/crio-6d2daf0443ba9fc80a0114cf348972073d4a7978dfc8b1315039881c009ad89e WatchSource:0}: Error finding container 6d2daf0443ba9fc80a0114cf348972073d4a7978dfc8b1315039881c009ad89e: Status 404 returned error can't find the container with id 6d2daf0443ba9fc80a0114cf348972073d4a7978dfc8b1315039881c009ad89e Jul 17 17:49:51.906829 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:51.906810 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6b07f4_0170_4df9_bcca_1e0613fe7079.slice/crio-a6735d2271e7f120e814992374a6a6407e71f1e4d5af1006c68b51c6908f7d31 WatchSource:0}: Error finding container a6735d2271e7f120e814992374a6a6407e71f1e4d5af1006c68b51c6908f7d31: Status 404 returned error can't find the container with id a6735d2271e7f120e814992374a6a6407e71f1e4d5af1006c68b51c6908f7d31 Jul 17 17:49:51.910147 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:51.910126 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8a7cf2_a598_4192_b51b_815e6ad11718.slice/crio-db2b1bc39c7bc3762b54cf0a7951de9234b031258d7900d66a1307689750141f WatchSource:0}: Error finding container db2b1bc39c7bc3762b54cf0a7951de9234b031258d7900d66a1307689750141f: Status 404 returned error can't find the container with id db2b1bc39c7bc3762b54cf0a7951de9234b031258d7900d66a1307689750141f Jul 17 17:49:51.911241 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:51.911222 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfe772e_cd4d_4f4b_92b3_49d68f4e4176.slice/crio-344e6444093746a1edc2c99add2ea56532b6ae383a14bffb10c5a231d069af2f WatchSource:0}: Error finding container 344e6444093746a1edc2c99add2ea56532b6ae383a14bffb10c5a231d069af2f: Status 404 returned error can't find the container with id 344e6444093746a1edc2c99add2ea56532b6ae383a14bffb10c5a231d069af2f Jul 17 17:49:51.912572 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:49:51.912550 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6a8a84_5e48_4b57_b683_a29619e9120f.slice/crio-17e441964e93b812f0c0cb6ed8764b95fbeb532b065b97b70e6df446529f9c7c WatchSource:0}: Error finding container 17e441964e93b812f0c0cb6ed8764b95fbeb532b065b97b70e6df446529f9c7c: Status 404 returned error can't find the container with id 17e441964e93b812f0c0cb6ed8764b95fbeb532b065b97b70e6df446529f9c7c Jul 17 17:49:51.993187 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.993020 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-db2n2" event={"ID":"3a6a8a84-5e48-4b57-b683-a29619e9120f","Type":"ContainerStarted","Data":"17e441964e93b812f0c0cb6ed8764b95fbeb532b065b97b70e6df446529f9c7c"} Jul 17 17:49:51.994723 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.994699 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" event={"ID":"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176","Type":"ContainerStarted","Data":"344e6444093746a1edc2c99add2ea56532b6ae383a14bffb10c5a231d069af2f"} Jul 17 17:49:51.995788 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.995769 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" event={"ID":"55537b77-ff5d-44be-b7f7-8001ab4dd01d","Type":"ContainerStarted","Data":"6d2daf0443ba9fc80a0114cf348972073d4a7978dfc8b1315039881c009ad89e"} Jul 17 17:49:51.996762 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:51.996744 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncm4s" event={"ID":"f044d914-e059-4c54-9ab3-b933c612b30a","Type":"ContainerStarted","Data":"c4fb53986d32022c7556e3326b43ccd192dd33388f8d144028b89d25698d253c"} Jul 17 17:49:52.002690 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:52.002672 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" event={"ID":"abf9fe12-d673-42ab-9178-66dd5a309513","Type":"ContainerStarted","Data":"6fdb81279079a31b66f994c250dbd0c9e3432e909f1a16fe3c08801785b2847e"} Jul 17 17:49:52.006316 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:52.006292 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-877hk" event={"ID":"e8015312-d14e-4a6b-a54d-8a73e8582aa5","Type":"ContainerStarted","Data":"d4efbc0f056ac41ebc1c93f7a2d0e43e82be385a8eef73cf031eb1cd477d1438"} Jul 17 17:49:52.010101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:52.010078 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sbkl7" event={"ID":"cd6b07f4-0170-4df9-bcca-1e0613fe7079","Type":"ContainerStarted","Data":"a6735d2271e7f120e814992374a6a6407e71f1e4d5af1006c68b51c6908f7d31"} Jul 17 17:49:52.014032 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:52.014005 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sfcxh" event={"ID":"db8a7cf2-a598-4192-b51b-815e6ad11718","Type":"ContainerStarted","Data":"db2b1bc39c7bc3762b54cf0a7951de9234b031258d7900d66a1307689750141f"} Jul 17 17:49:52.015579 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:52.015560 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hzh2t" event={"ID":"c0ed2480-e1bb-4002-bb74-b79be477ba8a","Type":"ContainerStarted","Data":"d7431f400b0f4e65ff8a40fb217ea09f94a1f3fd57dba9efae418faaea2f81ad"} Jul 17 17:49:52.569160 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:52.568174 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6tnn\" (UniqueName: \"kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn\") pod \"network-check-target-mp8l2\" (UID: \"6f28f659-4138-4e06-8674-b6e6f8ed3172\") " pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:49:52.569160 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:52.568227 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret\") pod \"global-pull-secret-syncer-4xjqt\" (UID: \"fa00821b-8e34-4030-9988-e37c6a12230d\") " pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:49:52.569160 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:52.568266 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs\") pod \"network-metrics-daemon-r8hm5\" (UID: \"315910cc-963f-4190-98a8-4453798d7fc7\") " pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:49:52.569160 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:52.568488 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:49:52.569160 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:52.568552 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs podName:315910cc-963f-4190-98a8-4453798d7fc7 nodeName:}" failed. No retries permitted until 2026-07-17 17:49:54.568533389 +0000 UTC m=+6.156486058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs") pod "network-metrics-daemon-r8hm5" (UID: "315910cc-963f-4190-98a8-4453798d7fc7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:49:52.569160 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:52.568943 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:49:52.569160 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:52.568959 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:49:52.569160 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:52.568971 2574 projected.go:196] Error preparing data for projected volume kube-api-access-t6tnn for pod openshift-network-diagnostics/network-check-target-mp8l2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:49:52.569160 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:52.569014 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn podName:6f28f659-4138-4e06-8674-b6e6f8ed3172 nodeName:}" failed. No retries permitted until 2026-07-17 17:49:54.569000488 +0000 UTC m=+6.156953167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t6tnn" (UniqueName: "kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn") pod "network-check-target-mp8l2" (UID: "6f28f659-4138-4e06-8674-b6e6f8ed3172") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:49:52.569160 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:52.569070 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:49:52.569160 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:52.569099 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret podName:fa00821b-8e34-4030-9988-e37c6a12230d nodeName:}" failed. No retries permitted until 2026-07-17 17:49:54.569087758 +0000 UTC m=+6.157040437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret") pod "global-pull-secret-syncer-4xjqt" (UID: "fa00821b-8e34-4030-9988-e37c6a12230d") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:49:52.885173 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:52.885022 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 17:44:50 +0000 UTC" deadline="2028-02-21 20:42:21.246591228 +0000 UTC" Jul 17 17:49:52.885173 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:52.885084 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14018h52m28.361511472s" Jul 17 17:49:52.987163 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:52.986408 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:49:52.987163 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:52.986530 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8hm5" podUID="315910cc-963f-4190-98a8-4453798d7fc7" Jul 17 17:49:52.987163 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:52.986896 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:49:52.987163 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:52.986976 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjqt" podUID="fa00821b-8e34-4030-9988-e37c6a12230d" Jul 17 17:49:52.987163 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:52.987027 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:49:52.987163 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:52.987085 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8l2" podUID="6f28f659-4138-4e06-8674-b6e6f8ed3172" Jul 17 17:49:53.030379 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:53.029880 2574 generic.go:356] "Generic (PLEG): container finished" podID="f271b5630fe33a388a47016c46496a70" containerID="69e94feba37a703de32f049f2b2f9f5289afd88aa814f9cad12351e1f4c019b9" exitCode=0 Jul 17 17:49:53.030379 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:53.029979 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-155.ec2.internal" event={"ID":"f271b5630fe33a388a47016c46496a70","Type":"ContainerDied","Data":"69e94feba37a703de32f049f2b2f9f5289afd88aa814f9cad12351e1f4c019b9"} Jul 17 17:49:53.045265 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:53.045235 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-155.ec2.internal" event={"ID":"7aea52351981b999e804455ac7eec0e6","Type":"ContainerStarted","Data":"412d31a4cffbff384c3d5ea3fda8768b9ef74368ca86fdad772f645fc031d982"} Jul 17 17:49:53.058195 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:53.058109 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-155.ec2.internal" podStartSLOduration=3.058088282 podStartE2EDuration="3.058088282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:49:53.057380352 +0000 UTC m=+4.645333039" watchObservedRunningTime="2026-07-17 17:49:53.058088282 +0000 UTC m=+4.646040970" Jul 17 17:49:54.060489 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:54.060451 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-155.ec2.internal" event={"ID":"f271b5630fe33a388a47016c46496a70","Type":"ContainerStarted","Data":"2530a14d8ff2f24efb6775770658fe1bdcba885a897b5585a2f01417784072f7"} Jul 17 17:49:54.585959 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:54.584882 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs\") pod \"network-metrics-daemon-r8hm5\" (UID: \"315910cc-963f-4190-98a8-4453798d7fc7\") " pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:49:54.585959 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:54.584970 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6tnn\" (UniqueName: \"kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn\") pod \"network-check-target-mp8l2\" (UID: \"6f28f659-4138-4e06-8674-b6e6f8ed3172\") " pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:49:54.585959 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:54.585003 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret\") pod \"global-pull-secret-syncer-4xjqt\" (UID: \"fa00821b-8e34-4030-9988-e37c6a12230d\") " pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:49:54.585959 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:54.585146 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:49:54.585959 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:54.585215 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret podName:fa00821b-8e34-4030-9988-e37c6a12230d nodeName:}" failed. No retries permitted until 2026-07-17 17:49:58.585195831 +0000 UTC m=+10.173148512 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret") pod "global-pull-secret-syncer-4xjqt" (UID: "fa00821b-8e34-4030-9988-e37c6a12230d") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:49:54.585959 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:54.585666 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:49:54.585959 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:54.585720 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs podName:315910cc-963f-4190-98a8-4453798d7fc7 nodeName:}" failed. No retries permitted until 2026-07-17 17:49:58.585703447 +0000 UTC m=+10.173656129 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs") pod "network-metrics-daemon-r8hm5" (UID: "315910cc-963f-4190-98a8-4453798d7fc7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:49:54.585959 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:54.585806 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:49:54.585959 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:54.585822 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:49:54.585959 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:54.585834 2574 projected.go:196] Error preparing data for projected volume kube-api-access-t6tnn for pod openshift-network-diagnostics/network-check-target-mp8l2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:49:54.585959 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:54.585871 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn podName:6f28f659-4138-4e06-8674-b6e6f8ed3172 nodeName:}" failed. No retries permitted until 2026-07-17 17:49:58.585859513 +0000 UTC m=+10.173812184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t6tnn" (UniqueName: "kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn") pod "network-check-target-mp8l2" (UID: "6f28f659-4138-4e06-8674-b6e6f8ed3172") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:49:54.979484 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:54.979451 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:49:54.979677 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:54.979465 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:49:54.979677 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:54.979583 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:49:54.979677 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:54.979588 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjqt" podUID="fa00821b-8e34-4030-9988-e37c6a12230d" Jul 17 17:49:54.979848 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:54.979675 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8l2" podUID="6f28f659-4138-4e06-8674-b6e6f8ed3172" Jul 17 17:49:54.979848 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:54.979776 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8hm5" podUID="315910cc-963f-4190-98a8-4453798d7fc7" Jul 17 17:49:56.979854 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:56.979822 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:49:56.980307 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:56.979954 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8l2" podUID="6f28f659-4138-4e06-8674-b6e6f8ed3172" Jul 17 17:49:56.980307 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:56.979829 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:49:56.980307 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:56.980052 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjqt" podUID="fa00821b-8e34-4030-9988-e37c6a12230d" Jul 17 17:49:56.980307 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:56.979829 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:49:56.980307 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:56.980139 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8hm5" podUID="315910cc-963f-4190-98a8-4453798d7fc7" Jul 17 17:49:58.621733 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:58.620937 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6tnn\" (UniqueName: \"kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn\") pod \"network-check-target-mp8l2\" (UID: \"6f28f659-4138-4e06-8674-b6e6f8ed3172\") " pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:49:58.621733 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:58.620986 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret\") pod \"global-pull-secret-syncer-4xjqt\" (UID: \"fa00821b-8e34-4030-9988-e37c6a12230d\") " pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:49:58.621733 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:58.621022 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs\") pod \"network-metrics-daemon-r8hm5\" (UID: \"315910cc-963f-4190-98a8-4453798d7fc7\") " pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:49:58.621733 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:58.621144 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:49:58.621733 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:58.621211 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs podName:315910cc-963f-4190-98a8-4453798d7fc7 nodeName:}" failed. No retries permitted until 2026-07-17 17:50:06.621190193 +0000 UTC m=+18.209142865 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs") pod "network-metrics-daemon-r8hm5" (UID: "315910cc-963f-4190-98a8-4453798d7fc7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:49:58.621733 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:58.621592 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:49:58.621733 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:58.621608 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:49:58.621733 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:58.621617 2574 projected.go:196] Error preparing data for projected volume kube-api-access-t6tnn for pod openshift-network-diagnostics/network-check-target-mp8l2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:49:58.621733 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:58.621651 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn podName:6f28f659-4138-4e06-8674-b6e6f8ed3172 nodeName:}" failed. No retries permitted until 2026-07-17 17:50:06.621639166 +0000 UTC m=+18.209591847 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t6tnn" (UniqueName: "kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn") pod "network-check-target-mp8l2" (UID: "6f28f659-4138-4e06-8674-b6e6f8ed3172") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:49:58.621733 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:58.621692 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:49:58.621733 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:58.621712 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret podName:fa00821b-8e34-4030-9988-e37c6a12230d nodeName:}" failed. No retries permitted until 2026-07-17 17:50:06.621705344 +0000 UTC m=+18.209658011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret") pod "global-pull-secret-syncer-4xjqt" (UID: "fa00821b-8e34-4030-9988-e37c6a12230d") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:49:58.980587 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:58.980555 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:49:58.980749 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:58.980652 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjqt" podUID="fa00821b-8e34-4030-9988-e37c6a12230d" Jul 17 17:49:58.980749 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:58.980703 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:49:58.980832 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:58.980745 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8l2" podUID="6f28f659-4138-4e06-8674-b6e6f8ed3172" Jul 17 17:49:58.980832 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:49:58.980771 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:49:58.980890 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:49:58.980843 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8hm5" podUID="315910cc-963f-4190-98a8-4453798d7fc7" Jul 17 17:50:00.981021 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:00.980984 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:50:00.981583 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:00.981162 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8hm5" podUID="315910cc-963f-4190-98a8-4453798d7fc7" Jul 17 17:50:00.981650 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:00.981636 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:50:00.981849 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:00.981797 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8l2" podUID="6f28f659-4138-4e06-8674-b6e6f8ed3172" Jul 17 17:50:00.981997 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:00.981986 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:50:00.982163 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:00.982150 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjqt" podUID="fa00821b-8e34-4030-9988-e37c6a12230d" Jul 17 17:50:02.979249 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:02.979211 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:50:02.979747 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:02.979218 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:50:02.979747 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:02.979346 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjqt" podUID="fa00821b-8e34-4030-9988-e37c6a12230d" Jul 17 17:50:02.979747 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:02.979401 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:50:02.979747 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:02.979492 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8l2" podUID="6f28f659-4138-4e06-8674-b6e6f8ed3172" Jul 17 17:50:02.979747 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:02.979582 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8hm5" podUID="315910cc-963f-4190-98a8-4453798d7fc7" Jul 17 17:50:04.980187 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:04.979662 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:50:04.980187 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:04.979663 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:50:04.980187 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:04.979806 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjqt" podUID="fa00821b-8e34-4030-9988-e37c6a12230d" Jul 17 17:50:04.980187 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:04.979674 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:50:04.980187 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:04.979915 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8l2" podUID="6f28f659-4138-4e06-8674-b6e6f8ed3172" Jul 17 17:50:04.980187 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:04.979975 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8hm5" podUID="315910cc-963f-4190-98a8-4453798d7fc7" Jul 17 17:50:06.684987 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:06.684931 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret\") pod \"global-pull-secret-syncer-4xjqt\" (UID: \"fa00821b-8e34-4030-9988-e37c6a12230d\") " pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:50:06.685578 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:06.685005 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs\") pod \"network-metrics-daemon-r8hm5\" (UID: \"315910cc-963f-4190-98a8-4453798d7fc7\") " pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:50:06.685578 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:06.685060 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6tnn\" (UniqueName: \"kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn\") pod \"network-check-target-mp8l2\" (UID: \"6f28f659-4138-4e06-8674-b6e6f8ed3172\") " pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:50:06.685578 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:06.685100 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:50:06.685578 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:06.685160 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:50:06.685578 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:06.685177 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret podName:fa00821b-8e34-4030-9988-e37c6a12230d nodeName:}" failed. No retries permitted until 2026-07-17 17:50:22.685148838 +0000 UTC m=+34.273101523 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret") pod "global-pull-secret-syncer-4xjqt" (UID: "fa00821b-8e34-4030-9988-e37c6a12230d") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:50:06.685578 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:06.685191 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:50:06.685578 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:06.685204 2574 projected.go:196] Error preparing data for projected volume kube-api-access-t6tnn for pod openshift-network-diagnostics/network-check-target-mp8l2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:50:06.685578 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:06.685207 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:50:06.685578 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:06.685238 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn podName:6f28f659-4138-4e06-8674-b6e6f8ed3172 nodeName:}" failed. No retries permitted until 2026-07-17 17:50:22.685228162 +0000 UTC m=+34.273180849 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t6tnn" (UniqueName: "kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn") pod "network-check-target-mp8l2" (UID: "6f28f659-4138-4e06-8674-b6e6f8ed3172") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:50:06.685578 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:06.685273 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs podName:315910cc-963f-4190-98a8-4453798d7fc7 nodeName:}" failed. No retries permitted until 2026-07-17 17:50:22.685253272 +0000 UTC m=+34.273205955 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs") pod "network-metrics-daemon-r8hm5" (UID: "315910cc-963f-4190-98a8-4453798d7fc7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:50:06.979594 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:06.979558 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:50:06.979772 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:06.979680 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8l2" podUID="6f28f659-4138-4e06-8674-b6e6f8ed3172" Jul 17 17:50:06.979772 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:06.979561 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:50:06.979772 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:06.979768 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8hm5" podUID="315910cc-963f-4190-98a8-4453798d7fc7" Jul 17 17:50:06.979905 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:06.979558 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:50:06.979905 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:06.979822 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjqt" podUID="fa00821b-8e34-4030-9988-e37c6a12230d" Jul 17 17:50:08.982115 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:08.982081 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:50:08.982600 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:08.982187 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8l2" podUID="6f28f659-4138-4e06-8674-b6e6f8ed3172" Jul 17 17:50:08.982600 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:08.982561 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:50:08.982712 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:08.982667 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8hm5" podUID="315910cc-963f-4190-98a8-4453798d7fc7" Jul 17 17:50:08.982763 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:08.982718 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:50:08.982813 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:08.982788 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjqt" podUID="fa00821b-8e34-4030-9988-e37c6a12230d" Jul 17 17:50:10.979315 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:10.979272 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:50:10.979780 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:10.979272 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:50:10.979780 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:10.979430 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8l2" podUID="6f28f659-4138-4e06-8674-b6e6f8ed3172" Jul 17 17:50:10.979780 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:10.979490 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjqt" podUID="fa00821b-8e34-4030-9988-e37c6a12230d" Jul 17 17:50:10.979780 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:10.979287 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:50:10.979780 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:10.979593 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8hm5" podUID="315910cc-963f-4190-98a8-4453798d7fc7" Jul 17 17:50:12.104002 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.103608 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sbkl7" event={"ID":"cd6b07f4-0170-4df9-bcca-1e0613fe7079","Type":"ContainerStarted","Data":"9b83eb16fb93aa9c62fc11af56e549f0c69b7a29f1046acfb4c504504199b7d7"} Jul 17 17:50:12.105039 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.105015 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sfcxh" event={"ID":"db8a7cf2-a598-4192-b51b-815e6ad11718","Type":"ContainerStarted","Data":"5b7258f8324093dd9492b942df5f660490ae275ea2eed938405770d834a94720"} Jul 17 17:50:12.106455 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.106430 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hzh2t" event={"ID":"c0ed2480-e1bb-4002-bb74-b79be477ba8a","Type":"ContainerStarted","Data":"c44e587cbe43807bbb45a0ab8c7cff01fc179a092ac9742377ef87c26adee002"} Jul 17 17:50:12.107738 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.107717 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-db2n2" event={"ID":"3a6a8a84-5e48-4b57-b683-a29619e9120f","Type":"ContainerStarted","Data":"5428adcffba6e9f81bfec2ea564a5f913ee62c881b7db7beba5aec37252598e8"} Jul 17 17:50:12.109088 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.109067 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" event={"ID":"bcfe772e-cd4d-4f4b-92b3-49d68f4e4176","Type":"ContainerStarted","Data":"db916181ad6ccfd4c78b675cb5e638ef2eb8283e6c902639e43cf8e52734e099"} Jul 17 17:50:12.111792 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.111773 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" event={"ID":"55537b77-ff5d-44be-b7f7-8001ab4dd01d","Type":"ContainerStarted","Data":"a6ba20be00af834d6317fee3cd2b4443b1ca6ab08b490b8899a6b9efe9b3a1a3"} Jul 17 17:50:12.111877 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.111836 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" event={"ID":"55537b77-ff5d-44be-b7f7-8001ab4dd01d","Type":"ContainerStarted","Data":"51ab18c212ec3d0f9dd297514f028913053f7dce7bda4fa205153dd21b505579"} Jul 17 17:50:12.111877 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.111868 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" event={"ID":"55537b77-ff5d-44be-b7f7-8001ab4dd01d","Type":"ContainerStarted","Data":"fe5c39168908985f03e97861590af9469746215b7b6ad6632aae42ef6d20fecd"} Jul 17 17:50:12.111964 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.111901 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" event={"ID":"55537b77-ff5d-44be-b7f7-8001ab4dd01d","Type":"ContainerStarted","Data":"dca1bfba551bafc8e8be483e8ce3fa5d216b6ba9d16ff7f30d3dae5c445d4b87"} Jul 17 17:50:12.111964 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.111936 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" event={"ID":"55537b77-ff5d-44be-b7f7-8001ab4dd01d","Type":"ContainerStarted","Data":"7f528685eedc40102a00fe4c694b543b0933c0cb67fa5ad1c8796d0ac9398dc1"} Jul 17 17:50:12.112056 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.111966 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" event={"ID":"55537b77-ff5d-44be-b7f7-8001ab4dd01d","Type":"ContainerStarted","Data":"5ccc652f04f5ef12c91c5c8cb1bfa63fa1020637052633bcbd68dd453c4503b9"} Jul 17 17:50:12.113073 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.113051 2574 generic.go:356] "Generic (PLEG): container finished" podID="f044d914-e059-4c54-9ab3-b933c612b30a" containerID="dc2b39929cd7b9fb01a005137e970ee3d66ca5dbfd032bd10fa5f359d43a2a64" exitCode=0 Jul 17 17:50:12.113164 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.113113 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncm4s" event={"ID":"f044d914-e059-4c54-9ab3-b933c612b30a","Type":"ContainerDied","Data":"dc2b39929cd7b9fb01a005137e970ee3d66ca5dbfd032bd10fa5f359d43a2a64"} Jul 17 17:50:12.114576 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.114543 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" event={"ID":"abf9fe12-d673-42ab-9178-66dd5a309513","Type":"ContainerStarted","Data":"a5f80c219ad31a92ae7c7baf8ede23830db90beb08de9edfeea941bb98d67f45"} Jul 17 17:50:12.118569 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.118530 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sbkl7" podStartSLOduration=4.973077225 podStartE2EDuration="24.118514356s" totalImagesPullingTime="19.145437131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:50:12.118295461 +0000 UTC m=+23.706248152" watchObservedRunningTime="2026-07-17 17:50:12.118514356 +0000 UTC m=+23.706467046" Jul 17 17:50:12.118995 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.118962 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-155.ec2.internal" podStartSLOduration=22.118954045 podStartE2EDuration="22.118954045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:49:54.078318384 +0000 UTC m=+5.666271075" watchObservedRunningTime="2026-07-17 17:50:12.118954045 +0000 UTC m=+23.706906734" Jul 17 17:50:12.132413 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.132371 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-db2n2" podStartSLOduration=4.990326246 podStartE2EDuration="24.132337625s" totalImagesPullingTime="19.142011379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:50:12.132069145 +0000 UTC m=+23.720021835" watchObservedRunningTime="2026-07-17 17:50:12.132337625 +0000 UTC m=+23.720290315" Jul 17 17:50:12.169885 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.169827 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7vcqg" podStartSLOduration=5.02735505 podStartE2EDuration="24.169806394s" totalImagesPullingTime="19.142451344s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:50:12.169484469 +0000 UTC m=+23.757437159" watchObservedRunningTime="2026-07-17 17:50:12.169806394 +0000 UTC m=+23.757759085" Jul 17 17:50:12.189087 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.189029 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hzh2t" podStartSLOduration=4.972580821 podStartE2EDuration="24.189009955s" totalImagesPullingTime="19.216429134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:50:12.188895508 +0000 UTC m=+23.776848200" watchObservedRunningTime="2026-07-17 17:50:12.189009955 +0000 UTC m=+23.776962647" Jul 17 17:50:12.203703 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.203637 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-sfcxh" podStartSLOduration=5.055006071 podStartE2EDuration="24.203620393s" totalImagesPullingTime="19.148614322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:50:12.203204176 +0000 UTC m=+23.791156866" watchObservedRunningTime="2026-07-17 17:50:12.203620393 +0000 UTC m=+23.791573082" Jul 17 17:50:12.488823 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.488788 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:12.889028 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.888992 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:12.892107 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.892081 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 17 17:50:12.931770 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.931594 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T17:50:12.892100395Z","UUID":"1c8df214-20c7-45b0-bde1-3890a2fd990b","Handler":null,"Name":"","Endpoint":""} Jul 17 17:50:12.933543 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.933517 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 17 17:50:12.933672 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.933564 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 17 17:50:12.980013 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.979961 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:50:12.980321 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.979971 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:50:12.980321 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:12.980096 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:50:12.980321 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:12.980090 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8l2" podUID="6f28f659-4138-4e06-8674-b6e6f8ed3172" Jul 17 17:50:12.980321 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:12.980195 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8hm5" podUID="315910cc-963f-4190-98a8-4453798d7fc7" Jul 17 17:50:12.980321 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:12.980283 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjqt" podUID="fa00821b-8e34-4030-9988-e37c6a12230d" Jul 17 17:50:13.009386 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:13.009279 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:13.111273 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:13.111232 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:13.119300 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:13.119256 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" event={"ID":"abf9fe12-d673-42ab-9178-66dd5a309513","Type":"ContainerStarted","Data":"8e61b1da5941bbb927c065b4f4fc5e4362caebd247ee8e14997febef007d6a27"} Jul 17 17:50:13.120815 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:13.120791 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-877hk" event={"ID":"e8015312-d14e-4a6b-a54d-8a73e8582aa5","Type":"ContainerStarted","Data":"cd373cfcdb0f11d84ef5bd336062882d4c9ffe0e7215317e60ce09d91fbb978b"} Jul 17 17:50:13.136989 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:13.136933 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-877hk" podStartSLOduration=5.982148614 podStartE2EDuration="25.136912929s" totalImagesPullingTime="19.154764315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:50:13.136343575 +0000 UTC m=+24.724296266" watchObservedRunningTime="2026-07-17 17:50:13.136912929 +0000 UTC m=+24.724865620" Jul 17 17:50:13.283598 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:13.283451 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:13.488122 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:13.488089 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:14.127625 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:14.127585 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" event={"ID":"55537b77-ff5d-44be-b7f7-8001ab4dd01d","Type":"ContainerStarted","Data":"ab567fb2772fa1aac27344ac6ebc042e63e3b02371d4788cc27c1f65fe774024"} Jul 17 17:50:14.712144 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:14.712063 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:14.883176 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:14.883001 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:14.980109 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:14.980075 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:50:14.980307 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:14.980118 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:50:14.980307 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:14.980208 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8l2" podUID="6f28f659-4138-4e06-8674-b6e6f8ed3172" Jul 17 17:50:14.980307 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:14.980277 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjqt" podUID="fa00821b-8e34-4030-9988-e37c6a12230d" Jul 17 17:50:14.980307 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:14.980288 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:50:14.980533 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:14.980412 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8hm5" podUID="315910cc-963f-4190-98a8-4453798d7fc7" Jul 17 17:50:15.091024 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:15.090989 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:15.132677 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:15.132638 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" event={"ID":"abf9fe12-d673-42ab-9178-66dd5a309513","Type":"ContainerStarted","Data":"d274f97f5932ab169da4e25637163499dff3832baf2b5e04721cce982d2edb2b"} Jul 17 17:50:15.152673 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:15.152627 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqjx6" podStartSLOduration=5.05375022 podStartE2EDuration="27.152610355s" totalImagesPullingTime="22.098860135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:49:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:50:15.152030919 +0000 UTC m=+26.739983600" watchObservedRunningTime="2026-07-17 17:50:15.152610355 +0000 UTC m=+26.740563044" Jul 17 17:50:15.268421 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:15.268308 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:15.320505 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:15.320472 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:15.485741 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:15.485710 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:16.491397 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:16.491346 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:16.682133 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:16.681939 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-db2n2" Jul 17 17:50:16.682569 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:16.682540 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-db2n2" Jul 17 17:50:16.887565 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:16.887532 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:16.979290 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:16.979253 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:50:16.979476 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:16.979255 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:50:16.979476 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:16.979396 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8l2" podUID="6f28f659-4138-4e06-8674-b6e6f8ed3172" Jul 17 17:50:16.979597 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:16.979472 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8hm5" podUID="315910cc-963f-4190-98a8-4453798d7fc7" Jul 17 17:50:16.979597 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:16.979253 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:50:16.979597 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:16.979552 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjqt" podUID="fa00821b-8e34-4030-9988-e37c6a12230d" Jul 17 17:50:17.086554 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:17.086465 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:17.140743 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:17.140706 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" event={"ID":"55537b77-ff5d-44be-b7f7-8001ab4dd01d","Type":"ContainerStarted","Data":"e6145c71ccfc8eefae86e23d187abefc9cb594b3ec509bd022f129df9a13e433"} Jul 17 17:50:17.140987 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:17.140970 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:50:17.141059 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:17.141036 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:50:17.142573 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:17.142546 2574 generic.go:356] "Generic (PLEG): container finished" podID="f044d914-e059-4c54-9ab3-b933c612b30a" containerID="89f66032f3e857ea88932cd8f9f5201bd76352fdf26366efd6b0ea58fea9d171" exitCode=0 Jul 17 17:50:17.142694 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:17.142659 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncm4s" event={"ID":"f044d914-e059-4c54-9ab3-b933c612b30a","Type":"ContainerDied","Data":"89f66032f3e857ea88932cd8f9f5201bd76352fdf26366efd6b0ea58fea9d171"} Jul 17 17:50:17.142838 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:17.142821 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-db2n2" Jul 17 17:50:17.143959 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:17.143256 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-db2n2" Jul 17 17:50:17.157681 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:17.157649 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:50:17.165730 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:17.165689 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" podStartSLOduration=9.949082077 podStartE2EDuration="29.165673939s" totalImagesPullingTime="19.216591862s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:50:17.163809255 +0000 UTC m=+28.751761945" watchObservedRunningTime="2026-07-17 17:50:17.165673939 +0000 UTC m=+28.753626628" Jul 17 17:50:17.285502 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:17.285468 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:17.484506 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:17.484467 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:18.145425 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:18.145393 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:50:18.161085 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:18.161061 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:50:18.981250 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:18.981208 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:50:18.981465 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:18.981260 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:50:18.981465 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:18.981334 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:50:18.981465 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:18.981341 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjqt" podUID="fa00821b-8e34-4030-9988-e37c6a12230d" Jul 17 17:50:18.981465 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:18.981448 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8hm5" podUID="315910cc-963f-4190-98a8-4453798d7fc7" Jul 17 17:50:18.981667 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:18.981531 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8l2" podUID="6f28f659-4138-4e06-8674-b6e6f8ed3172" Jul 17 17:50:19.149822 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:19.149787 2574 generic.go:356] "Generic (PLEG): container finished" podID="f044d914-e059-4c54-9ab3-b933c612b30a" containerID="c6eab9341311af1df5b4a12a4c0a0590a5fd12fe087682bb21b0161027dbb089" exitCode=0 Jul 17 17:50:19.150253 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:19.149877 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncm4s" event={"ID":"f044d914-e059-4c54-9ab3-b933c612b30a","Type":"ContainerDied","Data":"c6eab9341311af1df5b4a12a4c0a0590a5fd12fe087682bb21b0161027dbb089"} Jul 17 17:50:20.559462 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:20.559430 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:20.660582 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:20.660552 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:20.980052 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:20.980024 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:50:20.980210 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:20.980032 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:50:20.980210 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:20.980155 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjqt" podUID="fa00821b-8e34-4030-9988-e37c6a12230d" Jul 17 17:50:20.980332 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:20.980032 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:50:20.980332 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:20.980227 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8hm5" podUID="315910cc-963f-4190-98a8-4453798d7fc7" Jul 17 17:50:20.980332 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:20.980290 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8l2" podUID="6f28f659-4138-4e06-8674-b6e6f8ed3172" Jul 17 17:50:21.156293 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:21.156260 2574 generic.go:356] "Generic (PLEG): container finished" podID="f044d914-e059-4c54-9ab3-b933c612b30a" containerID="0b3e79125f35623c474387a34b128a6e2afe4d531f8460486502c6fd465ba5ef" exitCode=0 Jul 17 17:50:21.156480 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:21.156308 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncm4s" event={"ID":"f044d914-e059-4c54-9ab3-b933c612b30a","Type":"ContainerDied","Data":"0b3e79125f35623c474387a34b128a6e2afe4d531f8460486502c6fd465ba5ef"} Jul 17 17:50:22.704518 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:22.704487 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6tnn\" (UniqueName: \"kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn\") pod \"network-check-target-mp8l2\" (UID: \"6f28f659-4138-4e06-8674-b6e6f8ed3172\") " pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:50:22.704518 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:22.704525 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret\") pod \"global-pull-secret-syncer-4xjqt\" (UID: \"fa00821b-8e34-4030-9988-e37c6a12230d\") " pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:50:22.705114 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:22.704548 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs\") pod \"network-metrics-daemon-r8hm5\" (UID: \"315910cc-963f-4190-98a8-4453798d7fc7\") " pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:50:22.705114 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:22.704650 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 17:50:22.705114 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:22.704650 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:50:22.705114 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:22.704678 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 17:50:22.705114 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:22.704708 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 17:50:22.705114 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:22.704712 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs podName:315910cc-963f-4190-98a8-4453798d7fc7 nodeName:}" failed. No retries permitted until 2026-07-17 17:50:54.70469389 +0000 UTC m=+66.292646559 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs") pod "network-metrics-daemon-r8hm5" (UID: "315910cc-963f-4190-98a8-4453798d7fc7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 17:50:22.705114 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:22.704722 2574 projected.go:196] Error preparing data for projected volume kube-api-access-t6tnn for pod openshift-network-diagnostics/network-check-target-mp8l2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:50:22.705114 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:22.704729 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret podName:fa00821b-8e34-4030-9988-e37c6a12230d nodeName:}" failed. No retries permitted until 2026-07-17 17:50:54.704720975 +0000 UTC m=+66.292673643 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret") pod "global-pull-secret-syncer-4xjqt" (UID: "fa00821b-8e34-4030-9988-e37c6a12230d") : object "kube-system"/"original-pull-secret" not registered Jul 17 17:50:22.705114 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:22.704773 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn podName:6f28f659-4138-4e06-8674-b6e6f8ed3172 nodeName:}" failed. No retries permitted until 2026-07-17 17:50:54.704755537 +0000 UTC m=+66.292708222 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-t6tnn" (UniqueName: "kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn") pod "network-check-target-mp8l2" (UID: "6f28f659-4138-4e06-8674-b6e6f8ed3172") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 17:50:22.979519 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:22.979482 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:50:22.979715 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:22.979483 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:50:22.979715 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:22.979627 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8hm5" podUID="315910cc-963f-4190-98a8-4453798d7fc7" Jul 17 17:50:22.979715 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:22.979483 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:50:22.979715 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:22.979690 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8l2" podUID="6f28f659-4138-4e06-8674-b6e6f8ed3172" Jul 17 17:50:22.979905 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:22.979744 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjqt" podUID="fa00821b-8e34-4030-9988-e37c6a12230d" Jul 17 17:50:24.979692 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:24.979654 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:50:24.980172 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:24.979654 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:50:24.980172 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:24.979791 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjqt" podUID="fa00821b-8e34-4030-9988-e37c6a12230d" Jul 17 17:50:24.980172 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:24.979803 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:50:24.980172 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:24.979881 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8l2" podUID="6f28f659-4138-4e06-8674-b6e6f8ed3172" Jul 17 17:50:24.980172 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:24.979993 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8hm5" podUID="315910cc-963f-4190-98a8-4453798d7fc7" Jul 17 17:50:25.597111 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:25.597020 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:25.692463 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:25.692431 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:25.793727 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:25.793690 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:25.892284 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:25.892198 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:26.174157 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:26.174075 2574 generic.go:356] "Generic (PLEG): container finished" podID="f044d914-e059-4c54-9ab3-b933c612b30a" containerID="669ebe99a5a4c74f5c0ee208e699e15110d746ea1a08948de9201d9a4ad8b194" exitCode=0 Jul 17 17:50:26.174157 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:26.174144 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncm4s" event={"ID":"f044d914-e059-4c54-9ab3-b933c612b30a","Type":"ContainerDied","Data":"669ebe99a5a4c74f5c0ee208e699e15110d746ea1a08948de9201d9a4ad8b194"} Jul 17 17:50:26.443556 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:26.443290 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mp8l2"] Jul 17 17:50:26.443694 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:26.443620 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:50:26.443737 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:26.443708 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8l2" podUID="6f28f659-4138-4e06-8674-b6e6f8ed3172" Jul 17 17:50:26.446315 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:26.446280 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4xjqt"] Jul 17 17:50:26.446453 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:26.446414 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:50:26.446521 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:26.446503 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjqt" podUID="fa00821b-8e34-4030-9988-e37c6a12230d" Jul 17 17:50:26.447006 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:26.446982 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r8hm5"] Jul 17 17:50:26.447107 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:26.447097 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:50:26.447213 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:26.447184 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8hm5" podUID="315910cc-963f-4190-98a8-4453798d7fc7" Jul 17 17:50:27.179999 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:27.179967 2574 generic.go:356] "Generic (PLEG): container finished" podID="f044d914-e059-4c54-9ab3-b933c612b30a" containerID="3fd86d4168f35757ff204e5910624fd37ad82b49d0641c2b2a0488470774bb2f" exitCode=0 Jul 17 17:50:27.180619 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:27.180019 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncm4s" event={"ID":"f044d914-e059-4c54-9ab3-b933c612b30a","Type":"ContainerDied","Data":"3fd86d4168f35757ff204e5910624fd37ad82b49d0641c2b2a0488470774bb2f"} Jul 17 17:50:27.457473 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:27.457442 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:27.545086 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:27.545002 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:27.979259 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:27.979227 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:50:27.979259 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:27.979252 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:50:27.979489 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:27.979259 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:50:27.979489 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:27.979342 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8l2" podUID="6f28f659-4138-4e06-8674-b6e6f8ed3172" Jul 17 17:50:27.979489 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:27.979438 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8hm5" podUID="315910cc-963f-4190-98a8-4453798d7fc7" Jul 17 17:50:27.979582 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:27.979496 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjqt" podUID="fa00821b-8e34-4030-9988-e37c6a12230d" Jul 17 17:50:28.078820 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:28.078786 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:28.175015 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:28.174986 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:28.185748 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:28.185719 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ncm4s" event={"ID":"f044d914-e059-4c54-9ab3-b933c612b30a","Type":"ContainerStarted","Data":"4032f3d07509e6a92e57b0ca8118fbbb66e571bb6e5051f5a1ad71cfcc93ae61"} Jul 17 17:50:28.253183 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:28.253096 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:28.527619 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:28.527501 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:28.577286 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:28.577250 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:28.750700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:28.750667 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:29.326238 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:29.326203 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:29.426022 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:29.425916 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:29.736664 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:29.736636 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:29.796088 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:29.796057 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:29.979672 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:29.979643 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:50:29.979672 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:29.979671 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:50:29.979840 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:29.979643 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:50:29.979840 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:29.979764 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8hm5" podUID="315910cc-963f-4190-98a8-4453798d7fc7" Jul 17 17:50:29.979840 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:29.979815 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mp8l2" podUID="6f28f659-4138-4e06-8674-b6e6f8ed3172" Jul 17 17:50:29.979946 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:29.979884 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4xjqt" podUID="fa00821b-8e34-4030-9988-e37c6a12230d" Jul 17 17:50:30.531954 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.531923 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:30.761394 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.761346 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-155.ec2.internal" event="NodeReady" Jul 17 17:50:30.761550 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.761494 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 17 17:50:30.805933 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.805827 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ncm4s" podStartSLOduration=13.030211909 podStartE2EDuration="42.805810149s" totalImagesPullingTime="29.77559824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:49:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:50:28.211774134 +0000 UTC m=+39.799726836" watchObservedRunningTime="2026-07-17 17:50:30.805810149 +0000 UTC m=+42.393762838" Jul 17 17:50:30.806290 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.806272 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kpgqx"] Jul 17 17:50:30.829448 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.829415 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-86lxj"] Jul 17 17:50:30.829602 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.829525 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kpgqx" Jul 17 17:50:30.832051 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.832015 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 17 17:50:30.832051 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.832041 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-66fzz\"" Jul 17 17:50:30.832222 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.832064 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 17 17:50:30.845483 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.845302 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kpgqx"] Jul 17 17:50:30.845483 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.845347 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-86lxj"] Jul 17 17:50:30.845650 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.845497 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-86lxj" Jul 17 17:50:30.848119 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.848101 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 17 17:50:30.848221 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.848101 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s6zth\"" Jul 17 17:50:30.848398 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.848338 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 17 17:50:30.848524 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.848402 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 17 17:50:30.848524 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.848431 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 17 17:50:30.907572 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.907537 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-whhtb"] Jul 17 17:50:30.913466 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.913415 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:30.928640 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.928607 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-whhtb"] Jul 17 17:50:30.928775 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.928713 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-whhtb" Jul 17 17:50:30.931559 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.931533 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 17 17:50:30.931680 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.931582 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 17 17:50:30.931680 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.931593 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 17 17:50:30.931680 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.931593 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4fmcs\"" Jul 17 17:50:30.970772 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.970736 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7df87b83-ee0d-4171-a51f-311f238b59f8-metrics-tls\") pod \"dns-default-kpgqx\" (UID: \"7df87b83-ee0d-4171-a51f-311f238b59f8\") " pod="openshift-dns/dns-default-kpgqx" Jul 17 17:50:30.970772 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.970788 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlbxw\" (UniqueName: \"kubernetes.io/projected/b348a8ff-2cb1-4d03-83ce-cf149b3e1691-kube-api-access-nlbxw\") pod \"insights-runtime-extractor-86lxj\" (UID: \"b348a8ff-2cb1-4d03-83ce-cf149b3e1691\") " pod="openshift-insights/insights-runtime-extractor-86lxj" Jul 17 17:50:30.971020 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.970817 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b348a8ff-2cb1-4d03-83ce-cf149b3e1691-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-86lxj\" (UID: \"b348a8ff-2cb1-4d03-83ce-cf149b3e1691\") " pod="openshift-insights/insights-runtime-extractor-86lxj" Jul 17 17:50:30.971020 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.970859 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7df87b83-ee0d-4171-a51f-311f238b59f8-config-volume\") pod \"dns-default-kpgqx\" (UID: \"7df87b83-ee0d-4171-a51f-311f238b59f8\") " pod="openshift-dns/dns-default-kpgqx" Jul 17 17:50:30.971020 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.970973 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b348a8ff-2cb1-4d03-83ce-cf149b3e1691-crio-socket\") pod \"insights-runtime-extractor-86lxj\" (UID: \"b348a8ff-2cb1-4d03-83ce-cf149b3e1691\") " pod="openshift-insights/insights-runtime-extractor-86lxj" Jul 17 17:50:30.971020 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.971006 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b348a8ff-2cb1-4d03-83ce-cf149b3e1691-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-86lxj\" (UID: \"b348a8ff-2cb1-4d03-83ce-cf149b3e1691\") " pod="openshift-insights/insights-runtime-extractor-86lxj" Jul 17 17:50:30.971224 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.971032 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttrdh\" (UniqueName: \"kubernetes.io/projected/7df87b83-ee0d-4171-a51f-311f238b59f8-kube-api-access-ttrdh\") pod \"dns-default-kpgqx\" (UID: \"7df87b83-ee0d-4171-a51f-311f238b59f8\") " pod="openshift-dns/dns-default-kpgqx" Jul 17 17:50:30.971224 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.971106 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7df87b83-ee0d-4171-a51f-311f238b59f8-tmp-dir\") pod \"dns-default-kpgqx\" (UID: \"7df87b83-ee0d-4171-a51f-311f238b59f8\") " pod="openshift-dns/dns-default-kpgqx" Jul 17 17:50:30.971224 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.971181 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b348a8ff-2cb1-4d03-83ce-cf149b3e1691-data-volume\") pod \"insights-runtime-extractor-86lxj\" (UID: \"b348a8ff-2cb1-4d03-83ce-cf149b3e1691\") " pod="openshift-insights/insights-runtime-extractor-86lxj" Jul 17 17:50:30.971412 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:30.971392 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:31.072530 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.072433 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b348a8ff-2cb1-4d03-83ce-cf149b3e1691-crio-socket\") pod \"insights-runtime-extractor-86lxj\" (UID: \"b348a8ff-2cb1-4d03-83ce-cf149b3e1691\") " pod="openshift-insights/insights-runtime-extractor-86lxj" Jul 17 17:50:31.072530 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.072470 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b348a8ff-2cb1-4d03-83ce-cf149b3e1691-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-86lxj\" (UID: \"b348a8ff-2cb1-4d03-83ce-cf149b3e1691\") " pod="openshift-insights/insights-runtime-extractor-86lxj" Jul 17 17:50:31.072530 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.072488 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttrdh\" (UniqueName: \"kubernetes.io/projected/7df87b83-ee0d-4171-a51f-311f238b59f8-kube-api-access-ttrdh\") pod \"dns-default-kpgqx\" (UID: \"7df87b83-ee0d-4171-a51f-311f238b59f8\") " pod="openshift-dns/dns-default-kpgqx" Jul 17 17:50:31.072796 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.072552 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7df87b83-ee0d-4171-a51f-311f238b59f8-tmp-dir\") pod \"dns-default-kpgqx\" (UID: \"7df87b83-ee0d-4171-a51f-311f238b59f8\") " pod="openshift-dns/dns-default-kpgqx" Jul 17 17:50:31.072796 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.072619 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b348a8ff-2cb1-4d03-83ce-cf149b3e1691-data-volume\") pod \"insights-runtime-extractor-86lxj\" (UID: \"b348a8ff-2cb1-4d03-83ce-cf149b3e1691\") " pod="openshift-insights/insights-runtime-extractor-86lxj" Jul 17 17:50:31.072796 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.072649 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l428g\" (UniqueName: \"kubernetes.io/projected/2ff37b9c-1e7e-4daf-9b55-6bcd4c5d0261-kube-api-access-l428g\") pod \"ingress-canary-whhtb\" (UID: \"2ff37b9c-1e7e-4daf-9b55-6bcd4c5d0261\") " pod="openshift-ingress-canary/ingress-canary-whhtb" Jul 17 17:50:31.072796 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.072676 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7df87b83-ee0d-4171-a51f-311f238b59f8-metrics-tls\") pod \"dns-default-kpgqx\" (UID: \"7df87b83-ee0d-4171-a51f-311f238b59f8\") " pod="openshift-dns/dns-default-kpgqx" Jul 17 17:50:31.072796 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.072685 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b348a8ff-2cb1-4d03-83ce-cf149b3e1691-crio-socket\") pod \"insights-runtime-extractor-86lxj\" (UID: \"b348a8ff-2cb1-4d03-83ce-cf149b3e1691\") " pod="openshift-insights/insights-runtime-extractor-86lxj" Jul 17 17:50:31.072796 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.072702 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlbxw\" (UniqueName: \"kubernetes.io/projected/b348a8ff-2cb1-4d03-83ce-cf149b3e1691-kube-api-access-nlbxw\") pod \"insights-runtime-extractor-86lxj\" (UID: \"b348a8ff-2cb1-4d03-83ce-cf149b3e1691\") " pod="openshift-insights/insights-runtime-extractor-86lxj" Jul 17 17:50:31.072796 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.072732 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b348a8ff-2cb1-4d03-83ce-cf149b3e1691-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-86lxj\" (UID: \"b348a8ff-2cb1-4d03-83ce-cf149b3e1691\") " pod="openshift-insights/insights-runtime-extractor-86lxj" Jul 17 17:50:31.073094 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.072903 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ff37b9c-1e7e-4daf-9b55-6bcd4c5d0261-cert\") pod \"ingress-canary-whhtb\" (UID: \"2ff37b9c-1e7e-4daf-9b55-6bcd4c5d0261\") " pod="openshift-ingress-canary/ingress-canary-whhtb" Jul 17 17:50:31.073094 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.072927 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7df87b83-ee0d-4171-a51f-311f238b59f8-tmp-dir\") pod \"dns-default-kpgqx\" (UID: \"7df87b83-ee0d-4171-a51f-311f238b59f8\") " pod="openshift-dns/dns-default-kpgqx" Jul 17 17:50:31.073094 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.072948 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7df87b83-ee0d-4171-a51f-311f238b59f8-config-volume\") pod \"dns-default-kpgqx\" (UID: \"7df87b83-ee0d-4171-a51f-311f238b59f8\") " pod="openshift-dns/dns-default-kpgqx" Jul 17 17:50:31.073094 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.072984 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b348a8ff-2cb1-4d03-83ce-cf149b3e1691-data-volume\") pod \"insights-runtime-extractor-86lxj\" (UID: \"b348a8ff-2cb1-4d03-83ce-cf149b3e1691\") " pod="openshift-insights/insights-runtime-extractor-86lxj" Jul 17 17:50:31.073273 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.073222 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b348a8ff-2cb1-4d03-83ce-cf149b3e1691-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-86lxj\" (UID: \"b348a8ff-2cb1-4d03-83ce-cf149b3e1691\") " pod="openshift-insights/insights-runtime-extractor-86lxj" Jul 17 17:50:31.073348 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.073328 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7df87b83-ee0d-4171-a51f-311f238b59f8-config-volume\") pod \"dns-default-kpgqx\" (UID: \"7df87b83-ee0d-4171-a51f-311f238b59f8\") " pod="openshift-dns/dns-default-kpgqx" Jul 17 17:50:31.076846 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.076818 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7df87b83-ee0d-4171-a51f-311f238b59f8-metrics-tls\") pod \"dns-default-kpgqx\" (UID: \"7df87b83-ee0d-4171-a51f-311f238b59f8\") " pod="openshift-dns/dns-default-kpgqx" Jul 17 17:50:31.076846 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.076837 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b348a8ff-2cb1-4d03-83ce-cf149b3e1691-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-86lxj\" (UID: \"b348a8ff-2cb1-4d03-83ce-cf149b3e1691\") " pod="openshift-insights/insights-runtime-extractor-86lxj" Jul 17 17:50:31.080494 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.080472 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttrdh\" (UniqueName: \"kubernetes.io/projected/7df87b83-ee0d-4171-a51f-311f238b59f8-kube-api-access-ttrdh\") pod \"dns-default-kpgqx\" (UID: \"7df87b83-ee0d-4171-a51f-311f238b59f8\") " pod="openshift-dns/dns-default-kpgqx" Jul 17 17:50:31.080594 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.080518 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlbxw\" (UniqueName: \"kubernetes.io/projected/b348a8ff-2cb1-4d03-83ce-cf149b3e1691-kube-api-access-nlbxw\") pod \"insights-runtime-extractor-86lxj\" (UID: \"b348a8ff-2cb1-4d03-83ce-cf149b3e1691\") " pod="openshift-insights/insights-runtime-extractor-86lxj" Jul 17 17:50:31.138253 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.138213 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kpgqx" Jul 17 17:50:31.154409 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.154378 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-86lxj" Jul 17 17:50:31.174230 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.174051 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l428g\" (UniqueName: \"kubernetes.io/projected/2ff37b9c-1e7e-4daf-9b55-6bcd4c5d0261-kube-api-access-l428g\") pod \"ingress-canary-whhtb\" (UID: \"2ff37b9c-1e7e-4daf-9b55-6bcd4c5d0261\") " pod="openshift-ingress-canary/ingress-canary-whhtb" Jul 17 17:50:31.174370 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.174255 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ff37b9c-1e7e-4daf-9b55-6bcd4c5d0261-cert\") pod \"ingress-canary-whhtb\" (UID: \"2ff37b9c-1e7e-4daf-9b55-6bcd4c5d0261\") " pod="openshift-ingress-canary/ingress-canary-whhtb" Jul 17 17:50:31.177603 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.177573 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ff37b9c-1e7e-4daf-9b55-6bcd4c5d0261-cert\") pod \"ingress-canary-whhtb\" (UID: \"2ff37b9c-1e7e-4daf-9b55-6bcd4c5d0261\") " pod="openshift-ingress-canary/ingress-canary-whhtb" Jul 17 17:50:31.182251 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.182226 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l428g\" (UniqueName: \"kubernetes.io/projected/2ff37b9c-1e7e-4daf-9b55-6bcd4c5d0261-kube-api-access-l428g\") pod \"ingress-canary-whhtb\" (UID: \"2ff37b9c-1e7e-4daf-9b55-6bcd4c5d0261\") " pod="openshift-ingress-canary/ingress-canary-whhtb" Jul 17 17:50:31.237079 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.237040 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-whhtb" Jul 17 17:50:31.343667 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.343637 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kpgqx"] Jul 17 17:50:31.346835 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.346805 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-86lxj"] Jul 17 17:50:31.347798 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:50:31.347773 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df87b83_ee0d_4171_a51f_311f238b59f8.slice/crio-e96dce544f3abbaaf252f06a655855b07a30bcfac124afcf4983c284a432ec6f WatchSource:0}: Error finding container e96dce544f3abbaaf252f06a655855b07a30bcfac124afcf4983c284a432ec6f: Status 404 returned error can't find the container with id e96dce544f3abbaaf252f06a655855b07a30bcfac124afcf4983c284a432ec6f Jul 17 17:50:31.350230 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:50:31.350211 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb348a8ff_2cb1_4d03_83ce_cf149b3e1691.slice/crio-a3469a92404d3881ada71fc9f0392c4231c247f905e1f0a7360cf836beb7f382 WatchSource:0}: Error finding container a3469a92404d3881ada71fc9f0392c4231c247f905e1f0a7360cf836beb7f382: Status 404 returned error can't find the container with id a3469a92404d3881ada71fc9f0392c4231c247f905e1f0a7360cf836beb7f382 Jul 17 17:50:31.403509 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.403478 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-whhtb"] Jul 17 17:50:31.406500 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:50:31.406473 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff37b9c_1e7e_4daf_9b55_6bcd4c5d0261.slice/crio-98e4f42bcfc6515c51b3fba3bd083352d376d1cc2b1f4296e6ae0c57240ae6d2 WatchSource:0}: Error finding container 98e4f42bcfc6515c51b3fba3bd083352d376d1cc2b1f4296e6ae0c57240ae6d2: Status 404 returned error can't find the container with id 98e4f42bcfc6515c51b3fba3bd083352d376d1cc2b1f4296e6ae0c57240ae6d2 Jul 17 17:50:31.979575 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.979536 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:50:31.980078 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.979582 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:50:31.980078 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.979661 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:50:31.982519 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.982496 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 17:50:31.983675 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.983653 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 17:50:31.983776 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.983675 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xgrcl\"" Jul 17 17:50:31.983776 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.983742 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 17:50:31.983837 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.983658 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 17:50:31.983868 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:31.983653 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nlkcj\"" Jul 17 17:50:32.199098 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:32.199060 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-whhtb" event={"ID":"2ff37b9c-1e7e-4daf-9b55-6bcd4c5d0261","Type":"ContainerStarted","Data":"98e4f42bcfc6515c51b3fba3bd083352d376d1cc2b1f4296e6ae0c57240ae6d2"} Jul 17 17:50:32.201034 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:32.200925 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-86lxj" event={"ID":"b348a8ff-2cb1-4d03-83ce-cf149b3e1691","Type":"ContainerStarted","Data":"a3469a92404d3881ada71fc9f0392c4231c247f905e1f0a7360cf836beb7f382"} Jul 17 17:50:32.202389 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:32.202347 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kpgqx" event={"ID":"7df87b83-ee0d-4171-a51f-311f238b59f8","Type":"ContainerStarted","Data":"e96dce544f3abbaaf252f06a655855b07a30bcfac124afcf4983c284a432ec6f"} Jul 17 17:50:34.253810 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.253777 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:34.286039 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.286009 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-nglj8"] Jul 17 17:50:34.301797 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.301768 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-nglj8"] Jul 17 17:50:34.301986 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.301896 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-nglj8" Jul 17 17:50:34.305857 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.305837 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 17 17:50:34.306228 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.306206 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 17 17:50:34.306402 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.306302 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 17 17:50:34.307119 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.307099 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 17 17:50:34.307229 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.307106 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-x7dtk\"" Jul 17 17:50:34.307229 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.307192 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 17 17:50:34.328117 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.328088 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:34.401001 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.400961 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f34dcf01-3e4a-4d84-9e43-db829b532671-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-nglj8\" (UID: \"f34dcf01-3e4a-4d84-9e43-db829b532671\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nglj8" Jul 17 17:50:34.401201 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.401026 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f34dcf01-3e4a-4d84-9e43-db829b532671-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-nglj8\" (UID: \"f34dcf01-3e4a-4d84-9e43-db829b532671\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nglj8" Jul 17 17:50:34.401201 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.401132 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f34dcf01-3e4a-4d84-9e43-db829b532671-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-nglj8\" (UID: \"f34dcf01-3e4a-4d84-9e43-db829b532671\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nglj8" Jul 17 17:50:34.401201 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.401175 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfb4w\" (UniqueName: \"kubernetes.io/projected/f34dcf01-3e4a-4d84-9e43-db829b532671-kube-api-access-jfb4w\") pod \"prometheus-operator-5945b5cfd-nglj8\" (UID: \"f34dcf01-3e4a-4d84-9e43-db829b532671\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nglj8" Jul 17 17:50:34.502497 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.502455 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f34dcf01-3e4a-4d84-9e43-db829b532671-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-nglj8\" (UID: \"f34dcf01-3e4a-4d84-9e43-db829b532671\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nglj8" Jul 17 17:50:34.502687 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.502519 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f34dcf01-3e4a-4d84-9e43-db829b532671-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-nglj8\" (UID: \"f34dcf01-3e4a-4d84-9e43-db829b532671\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nglj8" Jul 17 17:50:34.502687 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.502592 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f34dcf01-3e4a-4d84-9e43-db829b532671-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-nglj8\" (UID: \"f34dcf01-3e4a-4d84-9e43-db829b532671\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nglj8" Jul 17 17:50:34.502687 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.502626 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfb4w\" (UniqueName: \"kubernetes.io/projected/f34dcf01-3e4a-4d84-9e43-db829b532671-kube-api-access-jfb4w\") pod \"prometheus-operator-5945b5cfd-nglj8\" (UID: \"f34dcf01-3e4a-4d84-9e43-db829b532671\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nglj8" Jul 17 17:50:34.502841 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:34.502765 2574 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 17 17:50:34.502896 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:34.502843 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f34dcf01-3e4a-4d84-9e43-db829b532671-prometheus-operator-tls podName:f34dcf01-3e4a-4d84-9e43-db829b532671 nodeName:}" failed. No retries permitted until 2026-07-17 17:50:35.00282345 +0000 UTC m=+46.590776142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/f34dcf01-3e4a-4d84-9e43-db829b532671-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-nglj8" (UID: "f34dcf01-3e4a-4d84-9e43-db829b532671") : secret "prometheus-operator-tls" not found Jul 17 17:50:34.514244 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.514176 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f34dcf01-3e4a-4d84-9e43-db829b532671-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-nglj8\" (UID: \"f34dcf01-3e4a-4d84-9e43-db829b532671\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nglj8" Jul 17 17:50:34.514244 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.514185 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f34dcf01-3e4a-4d84-9e43-db829b532671-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-nglj8\" (UID: \"f34dcf01-3e4a-4d84-9e43-db829b532671\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nglj8" Jul 17 17:50:34.528264 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.528229 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfb4w\" (UniqueName: \"kubernetes.io/projected/f34dcf01-3e4a-4d84-9e43-db829b532671-kube-api-access-jfb4w\") pod \"prometheus-operator-5945b5cfd-nglj8\" (UID: \"f34dcf01-3e4a-4d84-9e43-db829b532671\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nglj8" Jul 17 17:50:34.556299 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:34.556258 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:35.006726 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:35.006674 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f34dcf01-3e4a-4d84-9e43-db829b532671-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-nglj8\" (UID: \"f34dcf01-3e4a-4d84-9e43-db829b532671\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nglj8" Jul 17 17:50:35.009550 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:35.009512 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f34dcf01-3e4a-4d84-9e43-db829b532671-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-nglj8\" (UID: \"f34dcf01-3e4a-4d84-9e43-db829b532671\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nglj8" Jul 17 17:50:35.213801 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:35.213764 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-nglj8" Jul 17 17:50:37.611934 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:37.611715 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:37.706774 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:37.706694 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:38.404039 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:38.404012 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-nglj8"] Jul 17 17:50:38.491378 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:50:38.491335 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34dcf01_3e4a_4d84_9e43_db829b532671.slice/crio-4019103ce23d4755f47b1bff8b859765450ec51d2f70fa5075586426971c8b32 WatchSource:0}: Error finding container 4019103ce23d4755f47b1bff8b859765450ec51d2f70fa5075586426971c8b32: Status 404 returned error can't find the container with id 4019103ce23d4755f47b1bff8b859765450ec51d2f70fa5075586426971c8b32 Jul 17 17:50:39.224587 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:39.224291 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-86lxj" event={"ID":"b348a8ff-2cb1-4d03-83ce-cf149b3e1691","Type":"ContainerStarted","Data":"2b6fa6ae2bf296b3ab2b9b8cbc8998ed9895c12cbf9eb71a9db8989d04e3e5ef"} Jul 17 17:50:39.227181 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:39.227138 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kpgqx" event={"ID":"7df87b83-ee0d-4171-a51f-311f238b59f8","Type":"ContainerStarted","Data":"a4bafaf6951da36c6bcf718d70ee1b115fb69e85f17de51c95abd197a153efe8"} Jul 17 17:50:39.227327 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:39.227198 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kpgqx" event={"ID":"7df87b83-ee0d-4171-a51f-311f238b59f8","Type":"ContainerStarted","Data":"de46e7b4d2c3e5633739dd62fbd503790c6ee1e87999f06e9bed3e23465836f9"} Jul 17 17:50:39.227327 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:39.227258 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kpgqx" Jul 17 17:50:39.229264 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:39.229231 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-whhtb" event={"ID":"2ff37b9c-1e7e-4daf-9b55-6bcd4c5d0261","Type":"ContainerStarted","Data":"b9e524f7143286c22e8fad9f3f1f194caa7caaaf5f6ae459df35b8c0fa5ae4c6"} Jul 17 17:50:39.230784 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:39.230754 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-nglj8" event={"ID":"f34dcf01-3e4a-4d84-9e43-db829b532671","Type":"ContainerStarted","Data":"4019103ce23d4755f47b1bff8b859765450ec51d2f70fa5075586426971c8b32"} Jul 17 17:50:39.244463 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:39.244399 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kpgqx" podStartSLOduration=2.72620366 podStartE2EDuration="9.244383787s" totalImagesPullingTime="6.518180127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:50:39.243202425 +0000 UTC m=+50.831155136" watchObservedRunningTime="2026-07-17 17:50:39.244383787 +0000 UTC m=+50.832336474" Jul 17 17:50:39.258644 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:39.258575 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-whhtb" podStartSLOduration=2.802161896 podStartE2EDuration="9.258552746s" totalImagesPullingTime="6.45639085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:50:39.258174706 +0000 UTC m=+50.846127398" watchObservedRunningTime="2026-07-17 17:50:39.258552746 +0000 UTC m=+50.846505417" Jul 17 17:50:40.452294 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:40.452212 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:41.241494 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:41.241446 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-nglj8" event={"ID":"f34dcf01-3e4a-4d84-9e43-db829b532671","Type":"ContainerStarted","Data":"dd6d5b89c4939e808509ccd0e94297e5f6d6689938d2a49b430ae0a394c9ba42"} Jul 17 17:50:41.241753 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:41.241532 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-nglj8" event={"ID":"f34dcf01-3e4a-4d84-9e43-db829b532671","Type":"ContainerStarted","Data":"ea2d3679cacf93030cbb3f6fa315759f398c8d6b0edbe5e73aa7f9eb9fe8fcc9"} Jul 17 17:50:41.257529 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:41.257476 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-nglj8" podStartSLOduration=5.621522992 podStartE2EDuration="7.257454009s" totalImagesPullingTime="1.635931017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:50:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:50:41.256944854 +0000 UTC m=+52.844897567" watchObservedRunningTime="2026-07-17 17:50:41.257454009 +0000 UTC m=+52.845406701" Jul 17 17:50:41.495059 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:41.494977 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:41.595648 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:41.595610 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:42.655900 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:42.655860 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-b4dgk"] Jul 17 17:50:42.666791 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:42.666758 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:42.722993 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:42.722951 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-b4dgk"] Jul 17 17:50:42.723169 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:42.723076 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-b4dgk" Jul 17 17:50:42.725834 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:42.725812 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 17 17:50:42.725991 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:42.725968 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-77fhl\"" Jul 17 17:50:42.726048 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:42.726020 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 17 17:50:42.744029 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:42.743999 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:42.771638 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:42.771607 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsvfh\" (UniqueName: \"kubernetes.io/projected/b7866d5c-3eba-485c-9930-3b7823dc8cce-kube-api-access-lsvfh\") pod \"downloads-5bc67dcd46-b4dgk\" (UID: \"b7866d5c-3eba-485c-9930-3b7823dc8cce\") " pod="openshift-console/downloads-5bc67dcd46-b4dgk" Jul 17 17:50:42.804899 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:42.804877 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:42.872582 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:42.872543 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsvfh\" (UniqueName: \"kubernetes.io/projected/b7866d5c-3eba-485c-9930-3b7823dc8cce-kube-api-access-lsvfh\") pod \"downloads-5bc67dcd46-b4dgk\" (UID: \"b7866d5c-3eba-485c-9930-3b7823dc8cce\") " pod="openshift-console/downloads-5bc67dcd46-b4dgk" Jul 17 17:50:42.883106 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:42.883070 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsvfh\" (UniqueName: \"kubernetes.io/projected/b7866d5c-3eba-485c-9930-3b7823dc8cce-kube-api-access-lsvfh\") pod \"downloads-5bc67dcd46-b4dgk\" (UID: \"b7866d5c-3eba-485c-9930-3b7823dc8cce\") " pod="openshift-console/downloads-5bc67dcd46-b4dgk" Jul 17 17:50:42.932605 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:42.932529 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:43.034842 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.034798 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-b4dgk" Jul 17 17:50:43.061483 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.061439 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:43.162038 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.161863 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:43.180560 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.180531 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-b4dgk"] Jul 17 17:50:43.205814 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:50:43.205782 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7866d5c_3eba_485c_9930_3b7823dc8cce.slice/crio-72e13cb04d36b9119d70b7a534d4caa34653a4d03990384051285d445f508ec7 WatchSource:0}: Error finding container 72e13cb04d36b9119d70b7a534d4caa34653a4d03990384051285d445f508ec7: Status 404 returned error can't find the container with id 72e13cb04d36b9119d70b7a534d4caa34653a4d03990384051285d445f508ec7 Jul 17 17:50:43.250854 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.250828 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-b4dgk" event={"ID":"b7866d5c-3eba-485c-9930-3b7823dc8cce","Type":"ContainerStarted","Data":"72e13cb04d36b9119d70b7a534d4caa34653a4d03990384051285d445f508ec7"} Jul 17 17:50:43.303931 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.303900 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:43.404178 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.404144 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:43.508860 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.508798 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:43.607273 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.607238 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:43.610567 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.610542 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd"] Jul 17 17:50:43.626584 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.626555 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd"] Jul 17 17:50:43.626725 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.626687 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd" Jul 17 17:50:43.629145 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.629119 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 17 17:50:43.629289 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.629264 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 17 17:50:43.629457 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.629444 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-ms57f\"" Jul 17 17:50:43.635077 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.635054 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-hbxng"] Jul 17 17:50:43.648295 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.648266 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zdsq2"] Jul 17 17:50:43.648490 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.648407 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" Jul 17 17:50:43.652298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.652176 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 17 17:50:43.652468 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.652395 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xs7l7\"" Jul 17 17:50:43.652540 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.652499 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 17 17:50:43.652646 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.652622 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 17 17:50:43.659823 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.659800 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-hbxng"] Jul 17 17:50:43.660335 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.659924 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.662348 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.662328 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 17 17:50:43.662612 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.662588 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 17 17:50:43.662763 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.662601 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kp4nt\"" Jul 17 17:50:43.662763 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.662706 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 17 17:50:43.679025 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.678996 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e69c15b-cecf-46ca-99a7-4dd44d4b8c02-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-cj4hd\" (UID: \"1e69c15b-cecf-46ca-99a7-4dd44d4b8c02\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd" Jul 17 17:50:43.679193 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.679032 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59f60f6b-32d4-477b-a263-9c96b3eac478-node-exporter-wtmp\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.679193 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.679069 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e69c15b-cecf-46ca-99a7-4dd44d4b8c02-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-cj4hd\" (UID: \"1e69c15b-cecf-46ca-99a7-4dd44d4b8c02\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd" Jul 17 17:50:43.679193 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.679099 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59f60f6b-32d4-477b-a263-9c96b3eac478-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.679193 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.679120 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59f60f6b-32d4-477b-a263-9c96b3eac478-node-exporter-tls\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.679193 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.679173 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59f60f6b-32d4-477b-a263-9c96b3eac478-sys\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.679471 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.679208 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59f60f6b-32d4-477b-a263-9c96b3eac478-root\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.679471 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.679228 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59f60f6b-32d4-477b-a263-9c96b3eac478-node-exporter-accelerators-collector-config\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.679471 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.679265 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90507912-ab72-4d0a-927e-610d3b2ac868-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-hbxng\" (UID: \"90507912-ab72-4d0a-927e-610d3b2ac868\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" Jul 17 17:50:43.679471 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.679335 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lttdw\" (UniqueName: \"kubernetes.io/projected/90507912-ab72-4d0a-927e-610d3b2ac868-kube-api-access-lttdw\") pod \"kube-state-metrics-56c5594674-hbxng\" (UID: \"90507912-ab72-4d0a-927e-610d3b2ac868\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" Jul 17 17:50:43.679471 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.679385 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e69c15b-cecf-46ca-99a7-4dd44d4b8c02-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-cj4hd\" (UID: \"1e69c15b-cecf-46ca-99a7-4dd44d4b8c02\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd" Jul 17 17:50:43.679471 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.679412 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59f60f6b-32d4-477b-a263-9c96b3eac478-node-exporter-textfile\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.679471 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.679442 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqz9v\" (UniqueName: \"kubernetes.io/projected/59f60f6b-32d4-477b-a263-9c96b3eac478-kube-api-access-wqz9v\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.679471 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.679472 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/90507912-ab72-4d0a-927e-610d3b2ac868-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-hbxng\" (UID: \"90507912-ab72-4d0a-927e-610d3b2ac868\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" Jul 17 17:50:43.679736 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.679506 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90507912-ab72-4d0a-927e-610d3b2ac868-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-hbxng\" (UID: \"90507912-ab72-4d0a-927e-610d3b2ac868\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" Jul 17 17:50:43.679736 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.679538 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/90507912-ab72-4d0a-927e-610d3b2ac868-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-hbxng\" (UID: \"90507912-ab72-4d0a-927e-610d3b2ac868\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" Jul 17 17:50:43.679736 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.679563 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59f60f6b-32d4-477b-a263-9c96b3eac478-metrics-client-ca\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.679736 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.679583 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/90507912-ab72-4d0a-927e-610d3b2ac868-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-hbxng\" (UID: \"90507912-ab72-4d0a-927e-610d3b2ac868\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" Jul 17 17:50:43.679736 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.679598 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9flq9\" (UniqueName: \"kubernetes.io/projected/1e69c15b-cecf-46ca-99a7-4dd44d4b8c02-kube-api-access-9flq9\") pod \"openshift-state-metrics-79f6958cdf-cj4hd\" (UID: \"1e69c15b-cecf-46ca-99a7-4dd44d4b8c02\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd" Jul 17 17:50:43.692725 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.692693 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:43.781025 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.780935 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e69c15b-cecf-46ca-99a7-4dd44d4b8c02-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-cj4hd\" (UID: \"1e69c15b-cecf-46ca-99a7-4dd44d4b8c02\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd" Jul 17 17:50:43.781025 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.780988 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59f60f6b-32d4-477b-a263-9c96b3eac478-node-exporter-wtmp\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.781262 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.781029 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e69c15b-cecf-46ca-99a7-4dd44d4b8c02-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-cj4hd\" (UID: \"1e69c15b-cecf-46ca-99a7-4dd44d4b8c02\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd" Jul 17 17:50:43.781262 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.781064 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59f60f6b-32d4-477b-a263-9c96b3eac478-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.781262 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.781091 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59f60f6b-32d4-477b-a263-9c96b3eac478-node-exporter-tls\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.781262 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.781121 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59f60f6b-32d4-477b-a263-9c96b3eac478-sys\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.781262 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.781162 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59f60f6b-32d4-477b-a263-9c96b3eac478-root\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.781262 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.781176 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59f60f6b-32d4-477b-a263-9c96b3eac478-node-exporter-wtmp\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.781262 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.781187 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59f60f6b-32d4-477b-a263-9c96b3eac478-node-exporter-accelerators-collector-config\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.781262 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.781215 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90507912-ab72-4d0a-927e-610d3b2ac868-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-hbxng\" (UID: \"90507912-ab72-4d0a-927e-610d3b2ac868\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" Jul 17 17:50:43.781262 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.781248 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lttdw\" (UniqueName: \"kubernetes.io/projected/90507912-ab72-4d0a-927e-610d3b2ac868-kube-api-access-lttdw\") pod \"kube-state-metrics-56c5594674-hbxng\" (UID: \"90507912-ab72-4d0a-927e-610d3b2ac868\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" Jul 17 17:50:43.781262 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.781267 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e69c15b-cecf-46ca-99a7-4dd44d4b8c02-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-cj4hd\" (UID: \"1e69c15b-cecf-46ca-99a7-4dd44d4b8c02\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd" Jul 17 17:50:43.781620 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:43.781283 2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 17 17:50:43.781620 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:43.781341 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59f60f6b-32d4-477b-a263-9c96b3eac478-node-exporter-tls podName:59f60f6b-32d4-477b-a263-9c96b3eac478 nodeName:}" failed. No retries permitted until 2026-07-17 17:50:44.281322203 +0000 UTC m=+55.869274875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/59f60f6b-32d4-477b-a263-9c96b3eac478-node-exporter-tls") pod "node-exporter-zdsq2" (UID: "59f60f6b-32d4-477b-a263-9c96b3eac478") : secret "node-exporter-tls" not found Jul 17 17:50:43.781620 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.781580 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59f60f6b-32d4-477b-a263-9c96b3eac478-node-exporter-textfile\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.781716 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.781633 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59f60f6b-32d4-477b-a263-9c96b3eac478-sys\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.781716 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:43.781646 2574 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 17 17:50:43.781716 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.781661 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59f60f6b-32d4-477b-a263-9c96b3eac478-root\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.781716 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:43.781701 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e69c15b-cecf-46ca-99a7-4dd44d4b8c02-openshift-state-metrics-tls podName:1e69c15b-cecf-46ca-99a7-4dd44d4b8c02 nodeName:}" failed. No retries permitted until 2026-07-17 17:50:44.281682621 +0000 UTC m=+55.869635290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/1e69c15b-cecf-46ca-99a7-4dd44d4b8c02-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-cj4hd" (UID: "1e69c15b-cecf-46ca-99a7-4dd44d4b8c02") : secret "openshift-state-metrics-tls" not found Jul 17 17:50:43.781859 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.781288 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59f60f6b-32d4-477b-a263-9c96b3eac478-node-exporter-textfile\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.781905 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.781849 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqz9v\" (UniqueName: \"kubernetes.io/projected/59f60f6b-32d4-477b-a263-9c96b3eac478-kube-api-access-wqz9v\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.781905 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.781884 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/90507912-ab72-4d0a-927e-610d3b2ac868-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-hbxng\" (UID: \"90507912-ab72-4d0a-927e-610d3b2ac868\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" Jul 17 17:50:43.781998 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.781916 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90507912-ab72-4d0a-927e-610d3b2ac868-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-hbxng\" (UID: \"90507912-ab72-4d0a-927e-610d3b2ac868\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" Jul 17 17:50:43.781998 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.781945 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/90507912-ab72-4d0a-927e-610d3b2ac868-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-hbxng\" (UID: \"90507912-ab72-4d0a-927e-610d3b2ac868\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" Jul 17 17:50:43.781998 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.781974 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59f60f6b-32d4-477b-a263-9c96b3eac478-metrics-client-ca\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.782136 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.782002 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/90507912-ab72-4d0a-927e-610d3b2ac868-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-hbxng\" (UID: \"90507912-ab72-4d0a-927e-610d3b2ac868\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" Jul 17 17:50:43.782136 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.782026 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9flq9\" (UniqueName: \"kubernetes.io/projected/1e69c15b-cecf-46ca-99a7-4dd44d4b8c02-kube-api-access-9flq9\") pod \"openshift-state-metrics-79f6958cdf-cj4hd\" (UID: \"1e69c15b-cecf-46ca-99a7-4dd44d4b8c02\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd" Jul 17 17:50:43.782136 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.782115 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59f60f6b-32d4-477b-a263-9c96b3eac478-node-exporter-accelerators-collector-config\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.782876 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.782710 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e69c15b-cecf-46ca-99a7-4dd44d4b8c02-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-cj4hd\" (UID: \"1e69c15b-cecf-46ca-99a7-4dd44d4b8c02\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd" Jul 17 17:50:43.782876 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.782752 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/90507912-ab72-4d0a-927e-610d3b2ac868-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-hbxng\" (UID: \"90507912-ab72-4d0a-927e-610d3b2ac868\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" Jul 17 17:50:43.783224 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.783187 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/90507912-ab72-4d0a-927e-610d3b2ac868-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-hbxng\" (UID: \"90507912-ab72-4d0a-927e-610d3b2ac868\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" Jul 17 17:50:43.783899 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.783818 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59f60f6b-32d4-477b-a263-9c96b3eac478-metrics-client-ca\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.783899 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.783861 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90507912-ab72-4d0a-927e-610d3b2ac868-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-hbxng\" (UID: \"90507912-ab72-4d0a-927e-610d3b2ac868\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" Jul 17 17:50:43.786839 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.786789 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59f60f6b-32d4-477b-a263-9c96b3eac478-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.786839 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.786832 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90507912-ab72-4d0a-927e-610d3b2ac868-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-hbxng\" (UID: \"90507912-ab72-4d0a-927e-610d3b2ac868\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" Jul 17 17:50:43.786978 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.786889 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/90507912-ab72-4d0a-927e-610d3b2ac868-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-hbxng\" (UID: \"90507912-ab72-4d0a-927e-610d3b2ac868\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" Jul 17 17:50:43.788263 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.788216 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e69c15b-cecf-46ca-99a7-4dd44d4b8c02-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-cj4hd\" (UID: \"1e69c15b-cecf-46ca-99a7-4dd44d4b8c02\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd" Jul 17 17:50:43.790510 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.790458 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:43.793821 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.793796 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9flq9\" (UniqueName: \"kubernetes.io/projected/1e69c15b-cecf-46ca-99a7-4dd44d4b8c02-kube-api-access-9flq9\") pod \"openshift-state-metrics-79f6958cdf-cj4hd\" (UID: \"1e69c15b-cecf-46ca-99a7-4dd44d4b8c02\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd" Jul 17 17:50:43.794627 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.794435 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqz9v\" (UniqueName: \"kubernetes.io/projected/59f60f6b-32d4-477b-a263-9c96b3eac478-kube-api-access-wqz9v\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:43.795009 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.794990 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lttdw\" (UniqueName: \"kubernetes.io/projected/90507912-ab72-4d0a-927e-610d3b2ac868-kube-api-access-lttdw\") pod \"kube-state-metrics-56c5594674-hbxng\" (UID: \"90507912-ab72-4d0a-927e-610d3b2ac868\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" Jul 17 17:50:43.891259 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.891183 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:43.958600 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.958559 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" Jul 17 17:50:43.992787 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:43.992760 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:44.106848 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.106817 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:44.124961 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.124923 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-hbxng"] Jul 17 17:50:44.130224 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:50:44.130198 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90507912_ab72_4d0a_927e_610d3b2ac868.slice/crio-3f1f32364af8e376093751495296d682a94c8f834cc3b410a77a08c88e71a7df WatchSource:0}: Error finding container 3f1f32364af8e376093751495296d682a94c8f834cc3b410a77a08c88e71a7df: Status 404 returned error can't find the container with id 3f1f32364af8e376093751495296d682a94c8f834cc3b410a77a08c88e71a7df Jul 17 17:50:44.207698 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.207660 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:44.257566 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.257482 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-86lxj" event={"ID":"b348a8ff-2cb1-4d03-83ce-cf149b3e1691","Type":"ContainerStarted","Data":"77fe9799247db5b3dab5aface7d86250a0bd30790289da1b065988744f5c11aa"} Jul 17 17:50:44.258942 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.258906 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" event={"ID":"90507912-ab72-4d0a-927e-610d3b2ac868","Type":"ContainerStarted","Data":"3f1f32364af8e376093751495296d682a94c8f834cc3b410a77a08c88e71a7df"} Jul 17 17:50:44.290032 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.289986 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59f60f6b-32d4-477b-a263-9c96b3eac478-node-exporter-tls\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:44.290198 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.290147 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e69c15b-cecf-46ca-99a7-4dd44d4b8c02-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-cj4hd\" (UID: \"1e69c15b-cecf-46ca-99a7-4dd44d4b8c02\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd" Jul 17 17:50:44.293647 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.293619 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59f60f6b-32d4-477b-a263-9c96b3eac478-node-exporter-tls\") pod \"node-exporter-zdsq2\" (UID: \"59f60f6b-32d4-477b-a263-9c96b3eac478\") " pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:44.293809 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.293788 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e69c15b-cecf-46ca-99a7-4dd44d4b8c02-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-cj4hd\" (UID: \"1e69c15b-cecf-46ca-99a7-4dd44d4b8c02\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd" Jul 17 17:50:44.511175 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.511146 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:44.537250 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.537211 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd" Jul 17 17:50:44.570153 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.570103 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zdsq2" Jul 17 17:50:44.582262 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:50:44.582231 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f60f6b_32d4_477b_a263_9c96b3eac478.slice/crio-f4459748eff0968bc3fa2a0a371b2d379db29b55175d9a5b3a1f23e1964fca2c WatchSource:0}: Error finding container f4459748eff0968bc3fa2a0a371b2d379db29b55175d9a5b3a1f23e1964fca2c: Status 404 returned error can't find the container with id f4459748eff0968bc3fa2a0a371b2d379db29b55175d9a5b3a1f23e1964fca2c Jul 17 17:50:44.612166 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.612127 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:44.689963 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.689914 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:44.694394 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.693731 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:50:44.699803 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.697671 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.703672 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.702457 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 17 17:50:44.703672 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.702677 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 17 17:50:44.703672 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.702763 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 17 17:50:44.703672 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.702914 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 17 17:50:44.703672 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.702974 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 17 17:50:44.703672 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.703159 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 17 17:50:44.703672 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.703230 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 17 17:50:44.703672 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.703339 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 17 17:50:44.704494 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.704338 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-npbf6\"" Jul 17 17:50:44.704494 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.704388 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 17 17:50:44.707262 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.707241 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd"] Jul 17 17:50:44.716613 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.716522 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:50:44.725671 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:50:44.725640 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e69c15b_cecf_46ca_99a7_4dd44d4b8c02.slice/crio-bc2bee98a99001fe24fc530ef03efbe8ec29df16ce3b8daf6339f0c9de77e5be WatchSource:0}: Error finding container bc2bee98a99001fe24fc530ef03efbe8ec29df16ce3b8daf6339f0c9de77e5be: Status 404 returned error can't find the container with id bc2bee98a99001fe24fc530ef03efbe8ec29df16ce3b8daf6339f0c9de77e5be Jul 17 17:50:44.794385 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.794170 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.794489 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.794410 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8d9132c3-a362-42c1-a5fd-189ef1277034-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.794549 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.794492 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d9132c3-a362-42c1-a5fd-189ef1277034-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.794549 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.794532 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.794655 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.794552 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.794655 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.794628 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.794758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.794700 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d9132c3-a362-42c1-a5fd-189ef1277034-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.794758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.794719 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4hrh\" (UniqueName: \"kubernetes.io/projected/8d9132c3-a362-42c1-a5fd-189ef1277034-kube-api-access-h4hrh\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.794758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.794742 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d9132c3-a362-42c1-a5fd-189ef1277034-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.794906 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.794765 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-web-config\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.794906 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.794812 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-config-volume\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.794906 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.794834 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d9132c3-a362-42c1-a5fd-189ef1277034-config-out\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.794906 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.794849 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.838256 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.838177 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:44.896474 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.896417 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8d9132c3-a362-42c1-a5fd-189ef1277034-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.896627 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.896500 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d9132c3-a362-42c1-a5fd-189ef1277034-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.896627 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.896538 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.896627 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.896566 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.896627 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.896618 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.896832 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.896646 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d9132c3-a362-42c1-a5fd-189ef1277034-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.896832 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.896670 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4hrh\" (UniqueName: \"kubernetes.io/projected/8d9132c3-a362-42c1-a5fd-189ef1277034-kube-api-access-h4hrh\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.896832 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.896701 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d9132c3-a362-42c1-a5fd-189ef1277034-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.896832 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.896725 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-web-config\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.896832 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.896760 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-config-volume\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.896832 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.896781 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d9132c3-a362-42c1-a5fd-189ef1277034-config-out\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.896832 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.896803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.897265 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.896830 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.898118 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.898091 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d9132c3-a362-42c1-a5fd-189ef1277034-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.900728 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.900698 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8d9132c3-a362-42c1-a5fd-189ef1277034-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.902377 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:44.901147 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d9132c3-a362-42c1-a5fd-189ef1277034-alertmanager-trusted-ca-bundle podName:8d9132c3-a362-42c1-a5fd-189ef1277034 nodeName:}" failed. No retries permitted until 2026-07-17 17:50:45.401125169 +0000 UTC m=+56.989077865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/8d9132c3-a362-42c1-a5fd-189ef1277034-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "8d9132c3-a362-42c1-a5fd-189ef1277034") : configmap references non-existent config key: ca-bundle.crt Jul 17 17:50:44.911098 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.911042 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4hrh\" (UniqueName: \"kubernetes.io/projected/8d9132c3-a362-42c1-a5fd-189ef1277034-kube-api-access-h4hrh\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.914825 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.914785 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d9132c3-a362-42c1-a5fd-189ef1277034-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.918234 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.918177 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d9132c3-a362-42c1-a5fd-189ef1277034-config-out\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.918645 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.918575 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.920857 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.920798 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-config-volume\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.922864 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.922536 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.922864 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.922819 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.923864 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.923809 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.924868 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.924777 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-web-config\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.924868 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.924816 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:44.939550 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:44.939478 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:45.266437 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.266399 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zdsq2" event={"ID":"59f60f6b-32d4-477b-a263-9c96b3eac478","Type":"ContainerStarted","Data":"f4459748eff0968bc3fa2a0a371b2d379db29b55175d9a5b3a1f23e1964fca2c"} Jul 17 17:50:45.270000 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.269862 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd" event={"ID":"1e69c15b-cecf-46ca-99a7-4dd44d4b8c02","Type":"ContainerStarted","Data":"76c0b7009ebe6993ddf6f72454584ffb7c8d082e5e34f1fabc4598fbf73d7023"} Jul 17 17:50:45.270000 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.269927 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd" event={"ID":"1e69c15b-cecf-46ca-99a7-4dd44d4b8c02","Type":"ContainerStarted","Data":"af6ffe724dff0b82757d5dfaa8c9e348a79af029874d088fdaa6fd39889c6be5"} Jul 17 17:50:45.270000 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.269956 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd" event={"ID":"1e69c15b-cecf-46ca-99a7-4dd44d4b8c02","Type":"ContainerStarted","Data":"bc2bee98a99001fe24fc530ef03efbe8ec29df16ce3b8daf6339f0c9de77e5be"} Jul 17 17:50:45.402854 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.402800 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d9132c3-a362-42c1-a5fd-189ef1277034-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:45.403800 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.403744 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d9132c3-a362-42c1-a5fd-189ef1277034-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:45.609266 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.607321 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5f99569965-6w8tx"] Jul 17 17:50:45.612218 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.612189 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.615325 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.615300 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 17 17:50:45.615491 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.615333 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 17 17:50:45.615491 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.615301 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 17 17:50:45.615752 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.615730 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4noe11rlvkfgr\"" Jul 17 17:50:45.616525 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.616507 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 17 17:50:45.616613 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.616593 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 17 17:50:45.616847 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.616827 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7dg6t\"" Jul 17 17:50:45.622701 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.622678 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:50:45.630771 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.630742 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f99569965-6w8tx"] Jul 17 17:50:45.705346 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.705304 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/48125a20-2cbd-4d74-a8c1-7945053655e9-secret-grpc-tls\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.705346 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.705365 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/48125a20-2cbd-4d74-a8c1-7945053655e9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.705830 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.705438 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48125a20-2cbd-4d74-a8c1-7945053655e9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.705830 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.705501 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48125a20-2cbd-4d74-a8c1-7945053655e9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.705830 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.705531 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/48125a20-2cbd-4d74-a8c1-7945053655e9-secret-thanos-querier-tls\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.705830 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.705624 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48125a20-2cbd-4d74-a8c1-7945053655e9-metrics-client-ca\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.705830 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.705652 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/48125a20-2cbd-4d74-a8c1-7945053655e9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.705830 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.705677 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf6ts\" (UniqueName: \"kubernetes.io/projected/48125a20-2cbd-4d74-a8c1-7945053655e9-kube-api-access-zf6ts\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.808038 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.808004 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/48125a20-2cbd-4d74-a8c1-7945053655e9-secret-grpc-tls\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.808237 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.808050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/48125a20-2cbd-4d74-a8c1-7945053655e9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.808237 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.808085 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48125a20-2cbd-4d74-a8c1-7945053655e9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.808237 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.808122 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48125a20-2cbd-4d74-a8c1-7945053655e9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.808237 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.808150 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/48125a20-2cbd-4d74-a8c1-7945053655e9-secret-thanos-querier-tls\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.808237 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.808223 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48125a20-2cbd-4d74-a8c1-7945053655e9-metrics-client-ca\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.808477 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.808249 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/48125a20-2cbd-4d74-a8c1-7945053655e9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.808477 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.808271 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf6ts\" (UniqueName: \"kubernetes.io/projected/48125a20-2cbd-4d74-a8c1-7945053655e9-kube-api-access-zf6ts\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.809742 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.809647 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48125a20-2cbd-4d74-a8c1-7945053655e9-metrics-client-ca\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.813757 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.813701 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/48125a20-2cbd-4d74-a8c1-7945053655e9-secret-grpc-tls\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.813887 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.813788 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/48125a20-2cbd-4d74-a8c1-7945053655e9-secret-thanos-querier-tls\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.813957 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.813941 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/48125a20-2cbd-4d74-a8c1-7945053655e9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.814745 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.814688 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/48125a20-2cbd-4d74-a8c1-7945053655e9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.816957 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.816932 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/48125a20-2cbd-4d74-a8c1-7945053655e9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.818204 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.818180 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf6ts\" (UniqueName: \"kubernetes.io/projected/48125a20-2cbd-4d74-a8c1-7945053655e9-kube-api-access-zf6ts\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.818588 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.818559 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/48125a20-2cbd-4d74-a8c1-7945053655e9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f99569965-6w8tx\" (UID: \"48125a20-2cbd-4d74-a8c1-7945053655e9\") " pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.822123 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.822063 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:45.926254 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.926162 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:50:45.940396 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:45.940333 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:46.568163 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:46.568110 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:46.669256 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:46.668429 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:46.768214 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:46.768179 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:47.303231 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:47.303195 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:47.915212 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:47.915172 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-76c895f64f-2ps4v"] Jul 17 17:50:47.942963 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:47.942925 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76c895f64f-2ps4v"] Jul 17 17:50:47.943146 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:47.943062 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:47.945877 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:47.945840 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 17 17:50:47.946005 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:47.945962 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 17 17:50:47.947305 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:47.947288 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-lrl65\"" Jul 17 17:50:47.947446 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:47.947307 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 17 17:50:47.947446 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:47.947403 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 17 17:50:47.947674 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:47.947657 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3gevdt7f1fcb8\"" Jul 17 17:50:48.030534 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.030449 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76c895f64f-2ps4v\" (UID: \"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a\") " pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:48.030534 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.030493 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a-audit-log\") pod \"metrics-server-76c895f64f-2ps4v\" (UID: \"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a\") " pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:48.030696 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.030611 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a-metrics-server-audit-profiles\") pod \"metrics-server-76c895f64f-2ps4v\" (UID: \"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a\") " pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:48.030801 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.030694 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkbh9\" (UniqueName: \"kubernetes.io/projected/42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a-kube-api-access-fkbh9\") pod \"metrics-server-76c895f64f-2ps4v\" (UID: \"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a\") " pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:48.030801 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.030750 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a-secret-metrics-server-tls\") pod \"metrics-server-76c895f64f-2ps4v\" (UID: \"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a\") " pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:48.030865 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.030785 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a-secret-metrics-server-client-certs\") pod \"metrics-server-76c895f64f-2ps4v\" (UID: \"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a\") " pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:48.030865 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.030855 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a-client-ca-bundle\") pod \"metrics-server-76c895f64f-2ps4v\" (UID: \"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a\") " pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:48.140216 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.140190 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a-audit-log\") pod \"metrics-server-76c895f64f-2ps4v\" (UID: \"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a\") " pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:48.140309 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.140260 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a-metrics-server-audit-profiles\") pod \"metrics-server-76c895f64f-2ps4v\" (UID: \"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a\") " pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:48.140391 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.140317 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkbh9\" (UniqueName: \"kubernetes.io/projected/42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a-kube-api-access-fkbh9\") pod \"metrics-server-76c895f64f-2ps4v\" (UID: \"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a\") " pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:48.140391 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.140348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a-secret-metrics-server-tls\") pod \"metrics-server-76c895f64f-2ps4v\" (UID: \"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a\") " pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:48.140500 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.140400 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a-secret-metrics-server-client-certs\") pod \"metrics-server-76c895f64f-2ps4v\" (UID: \"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a\") " pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:48.140500 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.140450 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a-client-ca-bundle\") pod \"metrics-server-76c895f64f-2ps4v\" (UID: \"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a\") " pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:48.140599 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.140522 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76c895f64f-2ps4v\" (UID: \"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a\") " pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:48.141309 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.141273 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76c895f64f-2ps4v\" (UID: \"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a\") " pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:48.141946 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.141925 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a-audit-log\") pod \"metrics-server-76c895f64f-2ps4v\" (UID: \"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a\") " pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:48.142818 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.142795 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a-metrics-server-audit-profiles\") pod \"metrics-server-76c895f64f-2ps4v\" (UID: \"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a\") " pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:48.153999 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.152742 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkbh9\" (UniqueName: \"kubernetes.io/projected/42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a-kube-api-access-fkbh9\") pod \"metrics-server-76c895f64f-2ps4v\" (UID: \"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a\") " pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:48.153999 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.153115 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a-secret-metrics-server-tls\") pod \"metrics-server-76c895f64f-2ps4v\" (UID: \"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a\") " pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:48.153999 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.153839 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a-client-ca-bundle\") pod \"metrics-server-76c895f64f-2ps4v\" (UID: \"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a\") " pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:48.154179 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.154049 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a-secret-metrics-server-client-certs\") pod \"metrics-server-76c895f64f-2ps4v\" (UID: \"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a\") " pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:48.200039 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.199659 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f99569965-6w8tx"] Jul 17 17:50:48.211853 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:50:48.211820 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48125a20_2cbd_4d74_a8c1_7945053655e9.slice/crio-71c87e18d25f7964d14d50f6a0ead24648d430b31d0896451185551284260d37 WatchSource:0}: Error finding container 71c87e18d25f7964d14d50f6a0ead24648d430b31d0896451185551284260d37: Status 404 returned error can't find the container with id 71c87e18d25f7964d14d50f6a0ead24648d430b31d0896451185551284260d37 Jul 17 17:50:48.225948 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.225439 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:50:48.229229 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:50:48.229109 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9132c3_a362_42c1_a5fd_189ef1277034.slice/crio-3e6f10c3257cac318726c587cb60d1012980295a5873063a6b9830621f43ae94 WatchSource:0}: Error finding container 3e6f10c3257cac318726c587cb60d1012980295a5873063a6b9830621f43ae94: Status 404 returned error can't find the container with id 3e6f10c3257cac318726c587cb60d1012980295a5873063a6b9830621f43ae94 Jul 17 17:50:48.254208 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.253844 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:50:48.289191 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.288120 2574 generic.go:356] "Generic (PLEG): container finished" podID="59f60f6b-32d4-477b-a263-9c96b3eac478" containerID="e62982bcd8047f0c1a52a27dd0f0cd360ebcfb17f179d8f06f3085d43c26792b" exitCode=0 Jul 17 17:50:48.289191 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.288210 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zdsq2" event={"ID":"59f60f6b-32d4-477b-a263-9c96b3eac478","Type":"ContainerDied","Data":"e62982bcd8047f0c1a52a27dd0f0cd360ebcfb17f179d8f06f3085d43c26792b"} Jul 17 17:50:48.293765 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.293269 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" event={"ID":"48125a20-2cbd-4d74-a8c1-7945053655e9","Type":"ContainerStarted","Data":"71c87e18d25f7964d14d50f6a0ead24648d430b31d0896451185551284260d37"} Jul 17 17:50:48.303783 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.303182 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-86lxj" event={"ID":"b348a8ff-2cb1-4d03-83ce-cf149b3e1691","Type":"ContainerStarted","Data":"61cc416ed60a41fa859bd997540baa92a15db079617ee35725351be583a149e2"} Jul 17 17:50:48.307518 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.307425 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d9132c3-a362-42c1-a5fd-189ef1277034","Type":"ContainerStarted","Data":"3e6f10c3257cac318726c587cb60d1012980295a5873063a6b9830621f43ae94"} Jul 17 17:50:48.322652 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.321774 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" event={"ID":"90507912-ab72-4d0a-927e-610d3b2ac868","Type":"ContainerStarted","Data":"258347817a76c2cdc879e39c502520e2aa23dcf3c38dccbea6bda606391dda42"} Jul 17 17:50:48.322652 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.321843 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" event={"ID":"90507912-ab72-4d0a-927e-610d3b2ac868","Type":"ContainerStarted","Data":"34400b32c3e0714581c8d844a32cc3ad5736d1d6db43460771c9edbca3249eb7"} Jul 17 17:50:48.331648 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.331593 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-86lxj" podStartSLOduration=1.841571705 podStartE2EDuration="18.331570321s" totalImagesPullingTime="16.489998616s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:50:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:50:48.330012002 +0000 UTC m=+59.917964693" watchObservedRunningTime="2026-07-17 17:50:48.331570321 +0000 UTC m=+59.919523013" Jul 17 17:50:48.347372 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.346654 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cdbff84f-6w4hz"] Jul 17 17:50:48.354767 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.354699 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:50:48.363207 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.358554 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ml5rl\"" Jul 17 17:50:48.363207 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.360883 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 17 17:50:48.363207 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.361217 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 17 17:50:48.363207 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.361241 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 17 17:50:48.363207 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.361497 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 17 17:50:48.363207 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.361674 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 17 17:50:48.367678 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.367597 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cdbff84f-6w4hz"] Jul 17 17:50:48.406799 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.406696 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:48.424398 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.422372 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wcdrz"] Jul 17 17:50:48.427457 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.426643 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wcdrz" Jul 17 17:50:48.432631 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.431224 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-n85rb\"" Jul 17 17:50:48.432631 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.431312 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 17 17:50:48.438111 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.437513 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wcdrz"] Jul 17 17:50:48.443638 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.443399 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73507cce-1862-4050-a43c-d568330c5c59-oauth-serving-cert\") pod \"console-5cdbff84f-6w4hz\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:50:48.443638 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.443449 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73507cce-1862-4050-a43c-d568330c5c59-console-config\") pod \"console-5cdbff84f-6w4hz\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:50:48.443638 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.443482 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73507cce-1862-4050-a43c-d568330c5c59-console-oauth-config\") pod \"console-5cdbff84f-6w4hz\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:50:48.443638 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.443554 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73507cce-1862-4050-a43c-d568330c5c59-console-serving-cert\") pod \"console-5cdbff84f-6w4hz\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:50:48.443933 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.443660 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhfjf\" (UniqueName: \"kubernetes.io/projected/73507cce-1862-4050-a43c-d568330c5c59-kube-api-access-bhfjf\") pod \"console-5cdbff84f-6w4hz\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:50:48.443933 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.443720 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73507cce-1862-4050-a43c-d568330c5c59-service-ca\") pod \"console-5cdbff84f-6w4hz\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:50:48.458794 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.458760 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76c895f64f-2ps4v"] Jul 17 17:50:48.476081 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.476023 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:48.545226 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.545172 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73507cce-1862-4050-a43c-d568330c5c59-console-serving-cert\") pod \"console-5cdbff84f-6w4hz\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:50:48.545441 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.545251 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhfjf\" (UniqueName: \"kubernetes.io/projected/73507cce-1862-4050-a43c-d568330c5c59-kube-api-access-bhfjf\") pod \"console-5cdbff84f-6w4hz\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:50:48.545441 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.545319 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73507cce-1862-4050-a43c-d568330c5c59-service-ca\") pod \"console-5cdbff84f-6w4hz\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:50:48.545441 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.545389 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlmpv\" (UniqueName: \"kubernetes.io/projected/41c7de63-1216-4c58-ba85-53c3dfe70641-kube-api-access-xlmpv\") pod \"monitoring-plugin-7b7b7bc8bf-wcdrz\" (UID: \"41c7de63-1216-4c58-ba85-53c3dfe70641\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wcdrz" Jul 17 17:50:48.545441 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.545423 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/41c7de63-1216-4c58-ba85-53c3dfe70641-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-wcdrz\" (UID: \"41c7de63-1216-4c58-ba85-53c3dfe70641\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wcdrz" Jul 17 17:50:48.545676 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.545472 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73507cce-1862-4050-a43c-d568330c5c59-oauth-serving-cert\") pod \"console-5cdbff84f-6w4hz\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:50:48.545676 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.545505 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73507cce-1862-4050-a43c-d568330c5c59-console-config\") pod \"console-5cdbff84f-6w4hz\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:50:48.545676 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.545540 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73507cce-1862-4050-a43c-d568330c5c59-console-oauth-config\") pod \"console-5cdbff84f-6w4hz\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:50:48.547389 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.547267 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73507cce-1862-4050-a43c-d568330c5c59-oauth-serving-cert\") pod \"console-5cdbff84f-6w4hz\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:50:48.547389 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.547380 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73507cce-1862-4050-a43c-d568330c5c59-service-ca\") pod \"console-5cdbff84f-6w4hz\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:50:48.547576 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.547493 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73507cce-1862-4050-a43c-d568330c5c59-console-config\") pod \"console-5cdbff84f-6w4hz\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:50:48.548938 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.548894 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73507cce-1862-4050-a43c-d568330c5c59-console-oauth-config\") pod \"console-5cdbff84f-6w4hz\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:50:48.549803 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.549761 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73507cce-1862-4050-a43c-d568330c5c59-console-serving-cert\") pod \"console-5cdbff84f-6w4hz\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:50:48.556304 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.556277 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhfjf\" (UniqueName: \"kubernetes.io/projected/73507cce-1862-4050-a43c-d568330c5c59-kube-api-access-bhfjf\") pod \"console-5cdbff84f-6w4hz\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:50:48.646658 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.646617 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/41c7de63-1216-4c58-ba85-53c3dfe70641-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-wcdrz\" (UID: \"41c7de63-1216-4c58-ba85-53c3dfe70641\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wcdrz" Jul 17 17:50:48.646873 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.646795 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlmpv\" (UniqueName: \"kubernetes.io/projected/41c7de63-1216-4c58-ba85-53c3dfe70641-kube-api-access-xlmpv\") pod \"monitoring-plugin-7b7b7bc8bf-wcdrz\" (UID: \"41c7de63-1216-4c58-ba85-53c3dfe70641\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wcdrz" Jul 17 17:50:48.647228 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:48.647200 2574 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 17 17:50:48.647380 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:50:48.647269 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41c7de63-1216-4c58-ba85-53c3dfe70641-monitoring-plugin-cert podName:41c7de63-1216-4c58-ba85-53c3dfe70641 nodeName:}" failed. No retries permitted until 2026-07-17 17:50:49.147253896 +0000 UTC m=+60.735206564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/41c7de63-1216-4c58-ba85-53c3dfe70641-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-wcdrz" (UID: "41c7de63-1216-4c58-ba85-53c3dfe70641") : secret "monitoring-plugin-cert" not found Jul 17 17:50:48.656883 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.656848 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlmpv\" (UniqueName: \"kubernetes.io/projected/41c7de63-1216-4c58-ba85-53c3dfe70641-kube-api-access-xlmpv\") pod \"monitoring-plugin-7b7b7bc8bf-wcdrz\" (UID: \"41c7de63-1216-4c58-ba85-53c3dfe70641\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wcdrz" Jul 17 17:50:48.677790 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.677498 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:50:48.837915 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.837881 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6c4484c75-w6cpp"] Jul 17 17:50:48.843290 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.843259 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:48.847912 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.847882 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 17 17:50:48.848169 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.848150 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-68trj\"" Jul 17 17:50:48.848432 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.848414 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 17 17:50:48.848617 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.848601 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 17 17:50:48.848850 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.848835 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 17 17:50:48.849445 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.849418 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 17 17:50:48.869178 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.869131 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c4484c75-w6cpp"] Jul 17 17:50:48.876295 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.876259 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 17 17:50:48.926292 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.926255 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:48.953278 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.952537 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45700789-5a8a-4b85-a437-abdb1591e11e-serving-certs-ca-bundle\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:48.953278 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.952620 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45700789-5a8a-4b85-a437-abdb1591e11e-metrics-client-ca\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:48.953278 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.952671 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45700789-5a8a-4b85-a437-abdb1591e11e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:48.953278 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.952723 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v9bc\" (UniqueName: \"kubernetes.io/projected/45700789-5a8a-4b85-a437-abdb1591e11e-kube-api-access-9v9bc\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:48.953278 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.952762 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/45700789-5a8a-4b85-a437-abdb1591e11e-secret-telemeter-client\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:48.953278 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.952794 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/45700789-5a8a-4b85-a437-abdb1591e11e-telemeter-client-tls\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:48.953278 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.952824 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45700789-5a8a-4b85-a437-abdb1591e11e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:48.953278 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:48.952868 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/45700789-5a8a-4b85-a437-abdb1591e11e-federate-client-tls\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:49.053635 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.053600 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/45700789-5a8a-4b85-a437-abdb1591e11e-federate-client-tls\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:49.053808 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.053680 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45700789-5a8a-4b85-a437-abdb1591e11e-serving-certs-ca-bundle\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:49.053808 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.053758 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45700789-5a8a-4b85-a437-abdb1591e11e-metrics-client-ca\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:49.053934 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.053811 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45700789-5a8a-4b85-a437-abdb1591e11e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:49.053934 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.053863 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v9bc\" (UniqueName: \"kubernetes.io/projected/45700789-5a8a-4b85-a437-abdb1591e11e-kube-api-access-9v9bc\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:49.053934 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.053903 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/45700789-5a8a-4b85-a437-abdb1591e11e-secret-telemeter-client\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:49.054096 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.053940 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/45700789-5a8a-4b85-a437-abdb1591e11e-telemeter-client-tls\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:49.054096 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.053970 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45700789-5a8a-4b85-a437-abdb1591e11e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:49.056265 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.055067 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45700789-5a8a-4b85-a437-abdb1591e11e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:49.056265 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.055395 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45700789-5a8a-4b85-a437-abdb1591e11e-metrics-client-ca\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:49.056265 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.055943 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45700789-5a8a-4b85-a437-abdb1591e11e-serving-certs-ca-bundle\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:49.057389 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.057340 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/45700789-5a8a-4b85-a437-abdb1591e11e-federate-client-tls\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:49.058726 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.058681 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/45700789-5a8a-4b85-a437-abdb1591e11e-secret-telemeter-client\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:49.060684 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.060658 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/45700789-5a8a-4b85-a437-abdb1591e11e-telemeter-client-tls\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:49.061290 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.061265 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45700789-5a8a-4b85-a437-abdb1591e11e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:49.063680 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.063657 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v9bc\" (UniqueName: \"kubernetes.io/projected/45700789-5a8a-4b85-a437-abdb1591e11e-kube-api-access-9v9bc\") pod \"telemeter-client-6c4484c75-w6cpp\" (UID: \"45700789-5a8a-4b85-a437-abdb1591e11e\") " pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:49.155694 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.155215 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/41c7de63-1216-4c58-ba85-53c3dfe70641-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-wcdrz\" (UID: \"41c7de63-1216-4c58-ba85-53c3dfe70641\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wcdrz" Jul 17 17:50:49.160860 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.160794 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/41c7de63-1216-4c58-ba85-53c3dfe70641-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-wcdrz\" (UID: \"41c7de63-1216-4c58-ba85-53c3dfe70641\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wcdrz" Jul 17 17:50:49.175978 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.175734 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" Jul 17 17:50:49.238890 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.238858 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kpgqx" Jul 17 17:50:49.239105 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.239041 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cdbff84f-6w4hz"] Jul 17 17:50:49.244325 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:50:49.244299 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73507cce_1862_4050_a43c_d568330c5c59.slice/crio-c673e1e499a3940f6ab723b58e0dbc2af7efc21a6b446e345d8d2c5e29707667 WatchSource:0}: Error finding container c673e1e499a3940f6ab723b58e0dbc2af7efc21a6b446e345d8d2c5e29707667: Status 404 returned error can't find the container with id c673e1e499a3940f6ab723b58e0dbc2af7efc21a6b446e345d8d2c5e29707667 Jul 17 17:50:49.331834 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.331747 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" event={"ID":"90507912-ab72-4d0a-927e-610d3b2ac868","Type":"ContainerStarted","Data":"9e516eb681aaa66458e223cd17aed275253f9518f2134ad2a631d56ed310e286"} Jul 17 17:50:49.335744 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.334693 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zdsq2" event={"ID":"59f60f6b-32d4-477b-a263-9c96b3eac478","Type":"ContainerStarted","Data":"e48acdd4aa6f516b2f48b9af5890ed9d6dcaacf0e8bf931cc4282e1841c548e8"} Jul 17 17:50:49.335744 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.334753 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zdsq2" event={"ID":"59f60f6b-32d4-477b-a263-9c96b3eac478","Type":"ContainerStarted","Data":"2f51abfd325b936008122dde96bff92e69f4d361183526b22638eb0d0c845fd6"} Jul 17 17:50:49.338444 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.338411 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cdbff84f-6w4hz" event={"ID":"73507cce-1862-4050-a43c-d568330c5c59","Type":"ContainerStarted","Data":"c673e1e499a3940f6ab723b58e0dbc2af7efc21a6b446e345d8d2c5e29707667"} Jul 17 17:50:49.341185 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.341074 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:49.341720 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.341666 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" event={"ID":"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a","Type":"ContainerStarted","Data":"ccd88adce605813c30e8560beb95e21528411a349b8e8d3a87ce4d851964f852"} Jul 17 17:50:49.343420 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.343398 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-n85rb\"" Jul 17 17:50:49.351212 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.351023 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wcdrz" Jul 17 17:50:49.352961 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.352916 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-hbxng" podStartSLOduration=2.46717861 podStartE2EDuration="6.352898997s" totalImagesPullingTime="3.885720387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:50:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:50:49.351121926 +0000 UTC m=+60.939074618" watchObservedRunningTime="2026-07-17 17:50:49.352898997 +0000 UTC m=+60.940851687" Jul 17 17:50:49.364257 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.364220 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c4484c75-w6cpp"] Jul 17 17:50:49.371548 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.371037 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zdsq2" podStartSLOduration=2.831994265 podStartE2EDuration="6.371016729s" totalImagesPullingTime="3.431905955s" totalInitContainerRuntime="107.116509ms" isStatefulPod=true podCreationTimestamp="2026-07-17 17:50:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:50:49.369657544 +0000 UTC m=+60.957610238" watchObservedRunningTime="2026-07-17 17:50:49.371016729 +0000 UTC m=+60.958969420" Jul 17 17:50:49.599098 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.598953 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:49.699107 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.698966 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:49.757184 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.757082 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:49.904447 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.904343 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:50:49.915387 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.910025 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:49.934126 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.934090 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:50:49.934719 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.934697 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:49.941399 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.939453 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 17 17:50:49.941399 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.939689 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 17 17:50:49.941399 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.939861 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 17 17:50:49.941399 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.940124 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 17 17:50:49.941399 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.940306 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-b61sba05c5kus\"" Jul 17 17:50:49.941399 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.940496 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 17 17:50:49.941399 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.940663 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 17 17:50:49.941399 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.940828 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 17 17:50:49.945500 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.944685 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 17 17:50:49.945500 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.944719 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-n4gp9\"" Jul 17 17:50:49.945500 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.944864 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 17 17:50:49.945500 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.944994 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 17 17:50:49.945500 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.945057 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 17 17:50:49.945500 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.944994 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 17 17:50:49.945500 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.945097 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 17 17:50:49.950860 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:49.946999 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 17 17:50:50.068315 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.068281 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.068457 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.068322 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.068457 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.068343 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.068570 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.068464 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.068570 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.068500 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-web-config\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.068570 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.068532 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgcfh\" (UniqueName: \"kubernetes.io/projected/b978bdb1-82ea-40a0-b843-a148a14e5c22-kube-api-access-vgcfh\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.068570 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.068559 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.068757 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.068593 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-config\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.068757 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.068609 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.068757 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.068633 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b978bdb1-82ea-40a0-b843-a148a14e5c22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.068757 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.068662 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.068757 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.068676 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.068757 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.068710 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.068757 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.068736 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.068971 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.068771 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.068971 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.068821 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.068971 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.068861 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.068971 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.068885 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.068971 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.068918 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b978bdb1-82ea-40a0-b843-a148a14e5c22-config-out\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.068971 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.068943 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.171334 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.169559 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.171334 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.169608 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-web-config\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.171334 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.169637 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgcfh\" (UniqueName: \"kubernetes.io/projected/b978bdb1-82ea-40a0-b843-a148a14e5c22-kube-api-access-vgcfh\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.171334 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.169670 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.171334 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.169719 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-config\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.171334 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.169748 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.171334 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.169784 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b978bdb1-82ea-40a0-b843-a148a14e5c22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.171334 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.169828 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.171334 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.169853 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.171334 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.169905 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.171334 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.169935 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.171334 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.169969 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.171334 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.170053 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.171334 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.170150 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.171334 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.170179 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.171334 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.170216 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b978bdb1-82ea-40a0-b843-a148a14e5c22-config-out\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.171334 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.170242 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.172535 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.170277 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.172535 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.170302 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.172535 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.170333 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.172535 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.170646 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.172535 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.171602 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.172535 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.172057 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.175434 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.173440 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.175434 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.173822 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.176332 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.176305 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.177239 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.177214 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.179429 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.179274 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.181304 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.179718 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ddl85" Jul 17 17:50:50.182085 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.182041 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.182664 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.182626 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b978bdb1-82ea-40a0-b843-a148a14e5c22-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.184116 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.184093 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-config\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.185638 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.184699 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.185638 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.184834 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgcfh\" (UniqueName: \"kubernetes.io/projected/b978bdb1-82ea-40a0-b843-a148a14e5c22-kube-api-access-vgcfh\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.185638 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.185217 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.186299 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.186273 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.186944 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.186912 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-web-config\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.187551 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.187529 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.187641 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.187614 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.188548 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.188525 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.190658 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.190577 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b978bdb1-82ea-40a0-b843-a148a14e5c22-config-out\") pod \"prometheus-k8s-0\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.254530 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.254209 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:50:50.350389 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.350171 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" event={"ID":"45700789-5a8a-4b85-a437-abdb1591e11e","Type":"ContainerStarted","Data":"d83df2c8378a3afb82cc145c4c1366178ec66105c649c376bb7651b66589eb29"} Jul 17 17:50:50.352492 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.352448 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:50.406060 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.406029 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:50.561985 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.561955 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:50.622598 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.622552 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:50.974377 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:50.974333 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wcdrz"] Jul 17 17:50:51.112906 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:51.112820 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:51.167178 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:51.167118 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:51.498939 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:50:51.491700 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c7de63_1216_4c58_ba85_53c3dfe70641.slice/crio-dcb12e1d861b1075365106e584d7bcd2e9315c9a0f1250c54e6cfefa3c09bdf5 WatchSource:0}: Error finding container dcb12e1d861b1075365106e584d7bcd2e9315c9a0f1250c54e6cfefa3c09bdf5: Status 404 returned error can't find the container with id dcb12e1d861b1075365106e584d7bcd2e9315c9a0f1250c54e6cfefa3c09bdf5 Jul 17 17:50:51.701023 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:51.700976 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:50:52.123741 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:52.123649 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:52.174174 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:52.174140 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:52.365887 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:52.365846 2574 generic.go:356] "Generic (PLEG): container finished" podID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerID="b74c1974a28db1ea9df66c6c8f8701455780ad3d953a009c09ceb2e7b8154aa7" exitCode=0 Jul 17 17:50:52.366051 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:52.365944 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d9132c3-a362-42c1-a5fd-189ef1277034","Type":"ContainerDied","Data":"b74c1974a28db1ea9df66c6c8f8701455780ad3d953a009c09ceb2e7b8154aa7"} Jul 17 17:50:52.372421 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:52.372332 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wcdrz" event={"ID":"41c7de63-1216-4c58-ba85-53c3dfe70641","Type":"ContainerStarted","Data":"dcb12e1d861b1075365106e584d7bcd2e9315c9a0f1250c54e6cfefa3c09bdf5"} Jul 17 17:50:52.379663 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:52.379453 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" event={"ID":"48125a20-2cbd-4d74-a8c1-7945053655e9","Type":"ContainerStarted","Data":"d4c939a2ab68b93457bc152be980f98a912cd52c931b988c8851a8696a2b566c"} Jul 17 17:50:52.379663 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:52.379532 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" event={"ID":"48125a20-2cbd-4d74-a8c1-7945053655e9","Type":"ContainerStarted","Data":"899b2b05b63cd52be76afcd6434f2cdf50216c2392149d5f2475f380d835db85"} Jul 17 17:50:52.379663 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:52.379572 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" event={"ID":"48125a20-2cbd-4d74-a8c1-7945053655e9","Type":"ContainerStarted","Data":"4e3015eb89beccdaa5043d293a011676241397dea89f7d2320ff3d656424c8c9"} Jul 17 17:50:52.384005 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:52.383259 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" event={"ID":"42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a","Type":"ContainerStarted","Data":"32eaf3360984b70a97de521a3561a789863335da7c5fd87f49693cc7e9ee1416"} Jul 17 17:50:52.391549 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:52.391454 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd" event={"ID":"1e69c15b-cecf-46ca-99a7-4dd44d4b8c02","Type":"ContainerStarted","Data":"c4b6aafd34cb112b6a49bd8d9489fd7f7c4f5a1e0c7536f6d61e7677f2797368"} Jul 17 17:50:52.396804 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:52.396776 2574 generic.go:356] "Generic (PLEG): container finished" podID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerID="f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42" exitCode=0 Jul 17 17:50:52.396939 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:52.396848 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b978bdb1-82ea-40a0-b843-a148a14e5c22","Type":"ContainerDied","Data":"f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42"} Jul 17 17:50:52.396939 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:52.396920 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b978bdb1-82ea-40a0-b843-a148a14e5c22","Type":"ContainerStarted","Data":"edd1dc4ebac9094aa53f385001c5a9018ccd476789473becac2602cdf72df0b0"} Jul 17 17:50:52.411782 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:52.411722 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj4hd" podStartSLOduration=3.50408986 podStartE2EDuration="9.411697184s" totalImagesPullingTime="5.907607324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:50:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:50:52.410918384 +0000 UTC m=+63.998871074" watchObservedRunningTime="2026-07-17 17:50:52.411697184 +0000 UTC m=+63.999649875" Jul 17 17:50:52.432170 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:52.431893 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" podStartSLOduration=2.3346977349999998 podStartE2EDuration="5.431874054s" totalImagesPullingTime="3.097176319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:50:52.429994905 +0000 UTC m=+64.017947600" watchObservedRunningTime="2026-07-17 17:50:52.431874054 +0000 UTC m=+64.019826785" Jul 17 17:50:54.729670 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:54.729245 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret\") pod \"global-pull-secret-syncer-4xjqt\" (UID: \"fa00821b-8e34-4030-9988-e37c6a12230d\") " pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:50:54.729670 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:54.729313 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs\") pod \"network-metrics-daemon-r8hm5\" (UID: \"315910cc-963f-4190-98a8-4453798d7fc7\") " pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:50:54.729670 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:54.729404 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6tnn\" (UniqueName: \"kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn\") pod \"network-check-target-mp8l2\" (UID: \"6f28f659-4138-4e06-8674-b6e6f8ed3172\") " pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:50:54.733089 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:54.732748 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 17:50:54.733089 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:54.732964 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 17:50:54.734121 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:54.733972 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 17:50:54.743465 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:54.743438 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 17:50:54.744700 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:54.744670 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/315910cc-963f-4190-98a8-4453798d7fc7-metrics-certs\") pod \"network-metrics-daemon-r8hm5\" (UID: \"315910cc-963f-4190-98a8-4453798d7fc7\") " pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:50:54.747745 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:54.747693 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa00821b-8e34-4030-9988-e37c6a12230d-original-pull-secret\") pod \"global-pull-secret-syncer-4xjqt\" (UID: \"fa00821b-8e34-4030-9988-e37c6a12230d\") " pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:50:54.755168 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:54.755092 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6tnn\" (UniqueName: \"kubernetes.io/projected/6f28f659-4138-4e06-8674-b6e6f8ed3172-kube-api-access-t6tnn\") pod \"network-check-target-mp8l2\" (UID: \"6f28f659-4138-4e06-8674-b6e6f8ed3172\") " pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:50:54.791415 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:54.791373 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4xjqt" Jul 17 17:50:54.802738 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:54.802506 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xgrcl\"" Jul 17 17:50:54.808769 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:54.808557 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nlkcj\"" Jul 17 17:50:54.810649 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:54.810625 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8hm5" Jul 17 17:50:54.816123 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:54.816100 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:50:58.349672 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:58.349631 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:58.408948 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:58.408912 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:59.508567 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:59.508533 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:50:59.562282 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:50:59.562250 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:00.178327 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:00.178289 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:00.229136 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:00.229102 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:01.932652 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:01.932619 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:01.984599 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:01.984565 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:05.123661 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.123618 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mp8l2"] Jul 17 17:51:05.161641 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:51:05.161606 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f28f659_4138_4e06_8674_b6e6f8ed3172.slice/crio-a26010c47674598b96c48316d0f910bb138e794dfbb6db410b03a59718c6ade3 WatchSource:0}: Error finding container a26010c47674598b96c48316d0f910bb138e794dfbb6db410b03a59718c6ade3: Status 404 returned error can't find the container with id a26010c47674598b96c48316d0f910bb138e794dfbb6db410b03a59718c6ade3 Jul 17 17:51:05.358430 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.357895 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r8hm5"] Jul 17 17:51:05.361779 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.361638 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4xjqt"] Jul 17 17:51:05.365676 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:51:05.365573 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod315910cc_963f_4190_98a8_4453798d7fc7.slice/crio-3c162ae766dc979b026e76d2dd8d78b93ba06ac14022cdba5b14c2a9b30a90d3 WatchSource:0}: Error finding container 3c162ae766dc979b026e76d2dd8d78b93ba06ac14022cdba5b14c2a9b30a90d3: Status 404 returned error can't find the container with id 3c162ae766dc979b026e76d2dd8d78b93ba06ac14022cdba5b14c2a9b30a90d3 Jul 17 17:51:05.368833 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:51:05.368797 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa00821b_8e34_4030_9988_e37c6a12230d.slice/crio-5747b05a398ec54fa95040d537ca31a3a7eae62bc3651b3b11d0b411f00475c7 WatchSource:0}: Error finding container 5747b05a398ec54fa95040d537ca31a3a7eae62bc3651b3b11d0b411f00475c7: Status 404 returned error can't find the container with id 5747b05a398ec54fa95040d537ca31a3a7eae62bc3651b3b11d0b411f00475c7 Jul 17 17:51:05.477144 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.477056 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" event={"ID":"48125a20-2cbd-4d74-a8c1-7945053655e9","Type":"ContainerStarted","Data":"eabca3d29397c54974ae8a650e1ff84ed5fe3ffb69a1c8d746c099775941d8bb"} Jul 17 17:51:05.477322 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.477186 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" event={"ID":"48125a20-2cbd-4d74-a8c1-7945053655e9","Type":"ContainerStarted","Data":"ee47105cc232dbe623d8a6ab72e0c9eba4ce9858e87d36ed747e9f6c786dcb06"} Jul 17 17:51:05.477322 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.477235 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" event={"ID":"48125a20-2cbd-4d74-a8c1-7945053655e9","Type":"ContainerStarted","Data":"5104f4a16b9dd8b2d73a4c17b832600a46109ae76116ac431bd774e8f45f0f6d"} Jul 17 17:51:05.479709 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.479687 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:51:05.479932 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.479912 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r8hm5" event={"ID":"315910cc-963f-4190-98a8-4453798d7fc7","Type":"ContainerStarted","Data":"3c162ae766dc979b026e76d2dd8d78b93ba06ac14022cdba5b14c2a9b30a90d3"} Jul 17 17:51:05.482124 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.481886 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-b4dgk" event={"ID":"b7866d5c-3eba-485c-9930-3b7823dc8cce","Type":"ContainerStarted","Data":"3a93b8d477f92ba818b2f79dbccaa8ae18af0db6212b30d51f0fc3c9f33ea4af"} Jul 17 17:51:05.483394 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.482554 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-b4dgk" Jul 17 17:51:05.485433 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.485406 2574 patch_prober.go:28] interesting pod/downloads-5bc67dcd46-b4dgk container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.10:8080/\": dial tcp 10.134.0.10:8080: connect: connection refused" start-of-body= Jul 17 17:51:05.485621 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.485591 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-5bc67dcd46-b4dgk" podUID="b7866d5c-3eba-485c-9930-3b7823dc8cce" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.10:8080/\": dial tcp 10.134.0.10:8080: connect: connection refused" Jul 17 17:51:05.495902 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.495879 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" Jul 17 17:51:05.502249 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.501717 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b978bdb1-82ea-40a0-b843-a148a14e5c22","Type":"ContainerStarted","Data":"5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20"} Jul 17 17:51:05.502249 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.501797 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b978bdb1-82ea-40a0-b843-a148a14e5c22","Type":"ContainerStarted","Data":"4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7"} Jul 17 17:51:05.508095 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.505748 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wcdrz" event={"ID":"41c7de63-1216-4c58-ba85-53c3dfe70641","Type":"ContainerStarted","Data":"cc8424f5fce5418d992d7fd5de036665736bf87f4bb394e4144b41769dc2842c"} Jul 17 17:51:05.508095 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.507169 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wcdrz" Jul 17 17:51:05.511494 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.510341 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5f99569965-6w8tx" podStartSLOduration=4.138095047 podStartE2EDuration="20.510321316s" totalImagesPullingTime="16.372226269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:50:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:51:05.506971865 +0000 UTC m=+77.094924579" watchObservedRunningTime="2026-07-17 17:51:05.510321316 +0000 UTC m=+77.098274007" Jul 17 17:51:05.517287 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.516970 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" event={"ID":"45700789-5a8a-4b85-a437-abdb1591e11e","Type":"ContainerStarted","Data":"9f6087e6cd155856b8ce4d1e40a14ee25f8bb78b84bd5afafaa3fc598a3f91d8"} Jul 17 17:51:05.517287 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.517059 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" event={"ID":"45700789-5a8a-4b85-a437-abdb1591e11e","Type":"ContainerStarted","Data":"d4b8db7b60dc5e3984d23df2b4d260159dd72206407fc5927e09e7510b987c65"} Jul 17 17:51:05.517287 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.517093 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" event={"ID":"45700789-5a8a-4b85-a437-abdb1591e11e","Type":"ContainerStarted","Data":"58eafb6fc656c0dc3da22bb04918790078fe5bf7fac1ffc4cdfabcaef01af10d"} Jul 17 17:51:05.521280 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.521241 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wcdrz" Jul 17 17:51:05.521721 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.521695 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mp8l2" event={"ID":"6f28f659-4138-4e06-8674-b6e6f8ed3172","Type":"ContainerStarted","Data":"a26010c47674598b96c48316d0f910bb138e794dfbb6db410b03a59718c6ade3"} Jul 17 17:51:05.524016 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.523991 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cdbff84f-6w4hz" event={"ID":"73507cce-1862-4050-a43c-d568330c5c59","Type":"ContainerStarted","Data":"56775ed5b0883c2e85cd7df125f82216066e7923d8875324b59dc3c1e13abf96"} Jul 17 17:51:05.527554 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.527371 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-b4dgk" podStartSLOduration=1.5072906700000002 podStartE2EDuration="23.527333814s" totalImagesPullingTime="22.020043144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:51:05.525811285 +0000 UTC m=+77.113763975" watchObservedRunningTime="2026-07-17 17:51:05.527333814 +0000 UTC m=+77.115286505" Jul 17 17:51:05.528800 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.528652 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d9132c3-a362-42c1-a5fd-189ef1277034","Type":"ContainerStarted","Data":"fc59f95099605225b3aa67e76e3bb6fe8395f8db48f22428a8ede8f0a703833c"} Jul 17 17:51:05.528800 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.528731 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d9132c3-a362-42c1-a5fd-189ef1277034","Type":"ContainerStarted","Data":"525257b95d1069d04b432583562598f0cc005d2eb8b7e4676703957a3aa54dd0"} Jul 17 17:51:05.530348 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.530323 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4xjqt" event={"ID":"fa00821b-8e34-4030-9988-e37c6a12230d","Type":"ContainerStarted","Data":"5747b05a398ec54fa95040d537ca31a3a7eae62bc3651b3b11d0b411f00475c7"} Jul 17 17:51:05.568330 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.568119 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:05.588702 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.588163 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-wcdrz" podStartSLOduration=4.210791626 podStartE2EDuration="17.588140739s" totalImagesPullingTime="13.377349113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:51:05.58739935 +0000 UTC m=+77.175352045" watchObservedRunningTime="2026-07-17 17:51:05.588140739 +0000 UTC m=+77.176093430" Jul 17 17:51:05.609992 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.609426 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6c4484c75-w6cpp" podStartSLOduration=2.786131194 podStartE2EDuration="17.609403344s" totalImagesPullingTime="14.82327215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:51:05.608508409 +0000 UTC m=+77.196461121" watchObservedRunningTime="2026-07-17 17:51:05.609403344 +0000 UTC m=+77.197356037" Jul 17 17:51:05.626780 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.626709 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cdbff84f-6w4hz" podStartSLOduration=1.940812515 podStartE2EDuration="17.626689726s" totalImagesPullingTime="15.685877211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:51:05.626046695 +0000 UTC m=+77.213999413" watchObservedRunningTime="2026-07-17 17:51:05.626689726 +0000 UTC m=+77.214642416" Jul 17 17:51:05.635910 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:05.635767 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:06.577581 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:06.577542 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d9132c3-a362-42c1-a5fd-189ef1277034","Type":"ContainerStarted","Data":"d12b62d83719f065cefa7507464d95b2b02f41633d20fddefc9f49d2abb71d88"} Jul 17 17:51:06.578300 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:06.578276 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d9132c3-a362-42c1-a5fd-189ef1277034","Type":"ContainerStarted","Data":"d3a4afa436b78a87db2e95f567edd649283955f9e8e93d1a43a18f2520d90fc7"} Jul 17 17:51:06.578473 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:06.578456 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d9132c3-a362-42c1-a5fd-189ef1277034","Type":"ContainerStarted","Data":"9882987cca4122f37c4ecdd490dc7a0fc87467f981f2b0e072182641401d134b"} Jul 17 17:51:06.578631 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:06.578582 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d9132c3-a362-42c1-a5fd-189ef1277034","Type":"ContainerStarted","Data":"67c94ef00288b7720b677f562b743ef5e2dc705d3a5782981aaabb8b3b313c59"} Jul 17 17:51:06.591472 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:06.591440 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b978bdb1-82ea-40a0-b843-a148a14e5c22","Type":"ContainerStarted","Data":"c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25"} Jul 17 17:51:06.591793 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:06.591771 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b978bdb1-82ea-40a0-b843-a148a14e5c22","Type":"ContainerStarted","Data":"dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699"} Jul 17 17:51:06.591938 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:06.591926 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b978bdb1-82ea-40a0-b843-a148a14e5c22","Type":"ContainerStarted","Data":"6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031"} Jul 17 17:51:06.592061 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:06.592048 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b978bdb1-82ea-40a0-b843-a148a14e5c22","Type":"ContainerStarted","Data":"f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488"} Jul 17 17:51:06.606263 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:06.606211 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=6.713446792 podStartE2EDuration="22.60619412s" totalImagesPullingTime="15.833340101s" totalInitContainerRuntime="59.407227ms" isStatefulPod=false podCreationTimestamp="2026-07-17 17:50:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:51:06.605049051 +0000 UTC m=+78.193001747" watchObservedRunningTime="2026-07-17 17:51:06.60619412 +0000 UTC m=+78.194146811" Jul 17 17:51:06.623804 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:06.623771 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-b4dgk" Jul 17 17:51:06.639374 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:06.636817 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.065269047 podStartE2EDuration="17.636792574s" totalImagesPullingTime="12.541102953s" totalInitContainerRuntime="30.420574ms" isStatefulPod=false podCreationTimestamp="2026-07-17 17:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:51:06.635006834 +0000 UTC m=+78.222959525" watchObservedRunningTime="2026-07-17 17:51:06.636792574 +0000 UTC m=+78.224745264" Jul 17 17:51:08.255183 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:08.255114 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:51:08.255604 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:08.255419 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:51:08.612475 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:08.612382 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r8hm5" event={"ID":"315910cc-963f-4190-98a8-4453798d7fc7","Type":"ContainerStarted","Data":"f31c1c8c0ee4497d0b7640902132399a176265bed3459a90e19ab996fc49cee9"} Jul 17 17:51:08.612475 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:08.612442 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r8hm5" event={"ID":"315910cc-963f-4190-98a8-4453798d7fc7","Type":"ContainerStarted","Data":"5deb88b54cf4b0e6285f576cd88918d06e4112947f0c204d3360168e38a2b774"} Jul 17 17:51:08.678533 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:08.678496 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:51:08.679586 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:08.679559 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:51:08.685765 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:08.685663 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:51:08.707835 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:08.707788 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r8hm5" podStartSLOduration=78.139191139 podStartE2EDuration="1m20.707747891s" totalImagesPullingTime="2.568556752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:51:08.631895835 +0000 UTC m=+80.219848545" watchObservedRunningTime="2026-07-17 17:51:08.707747891 +0000 UTC m=+80.295700613" Jul 17 17:51:08.711898 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:08.711796 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:08.811621 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:08.811569 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:08.875444 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:08.863883 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:09.178060 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:09.177977 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cdbff84f-6w4hz"] Jul 17 17:51:09.340742 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:09.340578 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:09.396212 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:09.396182 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:09.624301 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:09.624270 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:51:09.735782 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:09.735756 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:09.788841 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:09.788665 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:10.255228 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:10.255193 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:51:10.414790 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:10.414736 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:11.519062 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:11.519015 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:11.570224 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:11.570181 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:13.646032 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:13.645833 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mp8l2" event={"ID":"6f28f659-4138-4e06-8674-b6e6f8ed3172","Type":"ContainerStarted","Data":"53ae59cdc82a32c5e6e0837959266425ee4eae67cc2b1ceae5010128c1132d54"} Jul 17 17:51:13.646032 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:13.646007 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:51:13.647617 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:13.647567 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4xjqt" event={"ID":"fa00821b-8e34-4030-9988-e37c6a12230d","Type":"ContainerStarted","Data":"48edd02ec757ea01837e6780530ffe15adc77a8d85a370b2948d21278161e0ce"} Jul 17 17:51:13.661284 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:13.661225 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mp8l2" podStartSLOduration=78.090113773 podStartE2EDuration="1m25.661207765s" totalImagesPullingTime="7.571093992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:51:13.660463359 +0000 UTC m=+85.248416071" watchObservedRunningTime="2026-07-17 17:51:13.661207765 +0000 UTC m=+85.249160458" Jul 17 17:51:13.680242 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:13.680188 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4xjqt" podStartSLOduration=77.289432617 podStartE2EDuration="1m24.680166545s" totalImagesPullingTime="7.390733928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 17:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:51:13.679345572 +0000 UTC m=+85.267298263" watchObservedRunningTime="2026-07-17 17:51:13.680166545 +0000 UTC m=+85.268119235" Jul 17 17:51:20.314694 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:20.314660 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:20.416096 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:20.416050 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:20.516593 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:20.516559 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:20.617036 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:20.616951 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:20.664185 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:20.664158 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:28.259662 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:28.259628 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:51:28.263887 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:28.263852 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-76c895f64f-2ps4v" Jul 17 17:51:35.656116 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:35.656051 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5cdbff84f-6w4hz" podUID="73507cce-1862-4050-a43c-d568330c5c59" containerName="console" containerID="cri-o://56775ed5b0883c2e85cd7df125f82216066e7923d8875324b59dc3c1e13abf96" gracePeriod=15 Jul 17 17:51:35.936318 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:35.936232 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:51:36.053776 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.053738 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73507cce-1862-4050-a43c-d568330c5c59-console-serving-cert\") pod \"73507cce-1862-4050-a43c-d568330c5c59\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " Jul 17 17:51:36.053776 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.053786 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73507cce-1862-4050-a43c-d568330c5c59-console-oauth-config\") pod \"73507cce-1862-4050-a43c-d568330c5c59\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " Jul 17 17:51:36.054034 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.053823 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73507cce-1862-4050-a43c-d568330c5c59-console-config\") pod \"73507cce-1862-4050-a43c-d568330c5c59\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " Jul 17 17:51:36.054034 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.053856 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73507cce-1862-4050-a43c-d568330c5c59-oauth-serving-cert\") pod \"73507cce-1862-4050-a43c-d568330c5c59\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " Jul 17 17:51:36.054034 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.053883 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73507cce-1862-4050-a43c-d568330c5c59-service-ca\") pod \"73507cce-1862-4050-a43c-d568330c5c59\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " Jul 17 17:51:36.054034 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.053913 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhfjf\" (UniqueName: \"kubernetes.io/projected/73507cce-1862-4050-a43c-d568330c5c59-kube-api-access-bhfjf\") pod \"73507cce-1862-4050-a43c-d568330c5c59\" (UID: \"73507cce-1862-4050-a43c-d568330c5c59\") " Jul 17 17:51:36.054311 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.054282 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73507cce-1862-4050-a43c-d568330c5c59-console-config" (OuterVolumeSpecName: "console-config") pod "73507cce-1862-4050-a43c-d568330c5c59" (UID: "73507cce-1862-4050-a43c-d568330c5c59"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:51:36.054448 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.054306 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73507cce-1862-4050-a43c-d568330c5c59-service-ca" (OuterVolumeSpecName: "service-ca") pod "73507cce-1862-4050-a43c-d568330c5c59" (UID: "73507cce-1862-4050-a43c-d568330c5c59"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:51:36.054448 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.054348 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73507cce-1862-4050-a43c-d568330c5c59-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "73507cce-1862-4050-a43c-d568330c5c59" (UID: "73507cce-1862-4050-a43c-d568330c5c59"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:51:36.056455 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.056425 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73507cce-1862-4050-a43c-d568330c5c59-kube-api-access-bhfjf" (OuterVolumeSpecName: "kube-api-access-bhfjf") pod "73507cce-1862-4050-a43c-d568330c5c59" (UID: "73507cce-1862-4050-a43c-d568330c5c59"). InnerVolumeSpecName "kube-api-access-bhfjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:51:36.056638 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.056621 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73507cce-1862-4050-a43c-d568330c5c59-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "73507cce-1862-4050-a43c-d568330c5c59" (UID: "73507cce-1862-4050-a43c-d568330c5c59"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:51:36.056772 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.056694 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73507cce-1862-4050-a43c-d568330c5c59-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "73507cce-1862-4050-a43c-d568330c5c59" (UID: "73507cce-1862-4050-a43c-d568330c5c59"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:51:36.154862 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.154824 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/73507cce-1862-4050-a43c-d568330c5c59-console-serving-cert\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:51:36.154862 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.154854 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/73507cce-1862-4050-a43c-d568330c5c59-console-oauth-config\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:51:36.154862 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.154866 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/73507cce-1862-4050-a43c-d568330c5c59-console-config\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:51:36.154862 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.154874 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/73507cce-1862-4050-a43c-d568330c5c59-oauth-serving-cert\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:51:36.155122 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.154883 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/73507cce-1862-4050-a43c-d568330c5c59-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:51:36.155122 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.154891 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhfjf\" (UniqueName: \"kubernetes.io/projected/73507cce-1862-4050-a43c-d568330c5c59-kube-api-access-bhfjf\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:51:36.763040 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.763002 2574 generic.go:356] "Generic (PLEG): container finished" podID="73507cce-1862-4050-a43c-d568330c5c59" containerID="56775ed5b0883c2e85cd7df125f82216066e7923d8875324b59dc3c1e13abf96" exitCode=0 Jul 17 17:51:36.763527 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.763050 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cdbff84f-6w4hz" Jul 17 17:51:36.763527 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.763101 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cdbff84f-6w4hz" event={"ID":"73507cce-1862-4050-a43c-d568330c5c59","Type":"ContainerDied","Data":"56775ed5b0883c2e85cd7df125f82216066e7923d8875324b59dc3c1e13abf96"} Jul 17 17:51:36.763527 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.763160 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cdbff84f-6w4hz" event={"ID":"73507cce-1862-4050-a43c-d568330c5c59","Type":"ContainerDied","Data":"c673e1e499a3940f6ab723b58e0dbc2af7efc21a6b446e345d8d2c5e29707667"} Jul 17 17:51:36.763527 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.763185 2574 scope.go:117] "RemoveContainer" containerID="56775ed5b0883c2e85cd7df125f82216066e7923d8875324b59dc3c1e13abf96" Jul 17 17:51:36.772586 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.772568 2574 scope.go:117] "RemoveContainer" containerID="56775ed5b0883c2e85cd7df125f82216066e7923d8875324b59dc3c1e13abf96" Jul 17 17:51:36.772905 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:51:36.772883 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56775ed5b0883c2e85cd7df125f82216066e7923d8875324b59dc3c1e13abf96\": container with ID starting with 56775ed5b0883c2e85cd7df125f82216066e7923d8875324b59dc3c1e13abf96 not found: ID does not exist" containerID="56775ed5b0883c2e85cd7df125f82216066e7923d8875324b59dc3c1e13abf96" Jul 17 17:51:36.772979 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.772914 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56775ed5b0883c2e85cd7df125f82216066e7923d8875324b59dc3c1e13abf96"} err="failed to get container status \"56775ed5b0883c2e85cd7df125f82216066e7923d8875324b59dc3c1e13abf96\": rpc error: code = NotFound desc = could not find container \"56775ed5b0883c2e85cd7df125f82216066e7923d8875324b59dc3c1e13abf96\": container with ID starting with 56775ed5b0883c2e85cd7df125f82216066e7923d8875324b59dc3c1e13abf96 not found: ID does not exist" Jul 17 17:51:36.783854 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.783820 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cdbff84f-6w4hz"] Jul 17 17:51:36.788074 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.788045 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cdbff84f-6w4hz"] Jul 17 17:51:36.986127 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:36.986093 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73507cce-1862-4050-a43c-d568330c5c59" path="/var/lib/kubelet/pods/73507cce-1862-4050-a43c-d568330c5c59/volumes" Jul 17 17:51:44.656341 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:44.656305 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mp8l2" Jul 17 17:51:50.255141 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:50.255097 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:51:50.310424 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:50.310397 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:51:50.837912 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:50.837882 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:51:56.118338 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:56.118301 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:59.705443 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:59.705404 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:51:59.754431 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:51:59.754400 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:52:03.944463 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:03.944427 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:52:03.945059 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:03.944816 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="alertmanager" containerID="cri-o://525257b95d1069d04b432583562598f0cc005d2eb8b7e4676703957a3aa54dd0" gracePeriod=120 Jul 17 17:52:03.945059 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:03.944875 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="kube-rbac-proxy-metric" containerID="cri-o://d3a4afa436b78a87db2e95f567edd649283955f9e8e93d1a43a18f2520d90fc7" gracePeriod=120 Jul 17 17:52:03.945059 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:03.944914 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="prom-label-proxy" containerID="cri-o://d12b62d83719f065cefa7507464d95b2b02f41633d20fddefc9f49d2abb71d88" gracePeriod=120 Jul 17 17:52:03.945059 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:03.944932 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="kube-rbac-proxy-web" containerID="cri-o://67c94ef00288b7720b677f562b743ef5e2dc705d3a5782981aaabb8b3b313c59" gracePeriod=120 Jul 17 17:52:03.945059 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:03.944949 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="kube-rbac-proxy" containerID="cri-o://9882987cca4122f37c4ecdd490dc7a0fc87467f981f2b0e072182641401d134b" gracePeriod=120 Jul 17 17:52:03.945059 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:03.944978 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="config-reloader" containerID="cri-o://fc59f95099605225b3aa67e76e3bb6fe8395f8db48f22428a8ede8f0a703833c" gracePeriod=120 Jul 17 17:52:04.881291 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:04.881254 2574 generic.go:356] "Generic (PLEG): container finished" podID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerID="d12b62d83719f065cefa7507464d95b2b02f41633d20fddefc9f49d2abb71d88" exitCode=0 Jul 17 17:52:04.881291 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:04.881280 2574 generic.go:356] "Generic (PLEG): container finished" podID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerID="d3a4afa436b78a87db2e95f567edd649283955f9e8e93d1a43a18f2520d90fc7" exitCode=0 Jul 17 17:52:04.881291 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:04.881288 2574 generic.go:356] "Generic (PLEG): container finished" podID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerID="9882987cca4122f37c4ecdd490dc7a0fc87467f981f2b0e072182641401d134b" exitCode=0 Jul 17 17:52:04.881291 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:04.881296 2574 generic.go:356] "Generic (PLEG): container finished" podID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerID="fc59f95099605225b3aa67e76e3bb6fe8395f8db48f22428a8ede8f0a703833c" exitCode=0 Jul 17 17:52:04.881291 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:04.881303 2574 generic.go:356] "Generic (PLEG): container finished" podID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerID="525257b95d1069d04b432583562598f0cc005d2eb8b7e4676703957a3aa54dd0" exitCode=0 Jul 17 17:52:04.881611 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:04.881321 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d9132c3-a362-42c1-a5fd-189ef1277034","Type":"ContainerDied","Data":"d12b62d83719f065cefa7507464d95b2b02f41633d20fddefc9f49d2abb71d88"} Jul 17 17:52:04.881611 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:04.881398 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d9132c3-a362-42c1-a5fd-189ef1277034","Type":"ContainerDied","Data":"d3a4afa436b78a87db2e95f567edd649283955f9e8e93d1a43a18f2520d90fc7"} Jul 17 17:52:04.881611 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:04.881426 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d9132c3-a362-42c1-a5fd-189ef1277034","Type":"ContainerDied","Data":"9882987cca4122f37c4ecdd490dc7a0fc87467f981f2b0e072182641401d134b"} Jul 17 17:52:04.881611 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:04.881453 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d9132c3-a362-42c1-a5fd-189ef1277034","Type":"ContainerDied","Data":"fc59f95099605225b3aa67e76e3bb6fe8395f8db48f22428a8ede8f0a703833c"} Jul 17 17:52:04.881611 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:04.881476 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d9132c3-a362-42c1-a5fd-189ef1277034","Type":"ContainerDied","Data":"525257b95d1069d04b432583562598f0cc005d2eb8b7e4676703957a3aa54dd0"} Jul 17 17:52:05.201411 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.201386 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:05.229551 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.229499 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-main-tls\") pod \"8d9132c3-a362-42c1-a5fd-189ef1277034\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " Jul 17 17:52:05.229749 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.229561 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d9132c3-a362-42c1-a5fd-189ef1277034-metrics-client-ca\") pod \"8d9132c3-a362-42c1-a5fd-189ef1277034\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " Jul 17 17:52:05.229749 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.229597 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-kube-rbac-proxy\") pod \"8d9132c3-a362-42c1-a5fd-189ef1277034\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " Jul 17 17:52:05.229749 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.229624 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d9132c3-a362-42c1-a5fd-189ef1277034-config-out\") pod \"8d9132c3-a362-42c1-a5fd-189ef1277034\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " Jul 17 17:52:05.229749 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.229657 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-kube-rbac-proxy-metric\") pod \"8d9132c3-a362-42c1-a5fd-189ef1277034\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " Jul 17 17:52:05.229749 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.229685 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8d9132c3-a362-42c1-a5fd-189ef1277034-alertmanager-main-db\") pod \"8d9132c3-a362-42c1-a5fd-189ef1277034\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " Jul 17 17:52:05.229749 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.229708 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-kube-rbac-proxy-web\") pod \"8d9132c3-a362-42c1-a5fd-189ef1277034\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " Jul 17 17:52:05.230049 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.229751 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d9132c3-a362-42c1-a5fd-189ef1277034-alertmanager-trusted-ca-bundle\") pod \"8d9132c3-a362-42c1-a5fd-189ef1277034\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " Jul 17 17:52:05.230049 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.229790 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-config-volume\") pod \"8d9132c3-a362-42c1-a5fd-189ef1277034\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " Jul 17 17:52:05.230049 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.229851 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4hrh\" (UniqueName: \"kubernetes.io/projected/8d9132c3-a362-42c1-a5fd-189ef1277034-kube-api-access-h4hrh\") pod \"8d9132c3-a362-42c1-a5fd-189ef1277034\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " Jul 17 17:52:05.230049 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.229887 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d9132c3-a362-42c1-a5fd-189ef1277034-tls-assets\") pod \"8d9132c3-a362-42c1-a5fd-189ef1277034\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " Jul 17 17:52:05.230049 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.229939 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-web-config\") pod \"8d9132c3-a362-42c1-a5fd-189ef1277034\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " Jul 17 17:52:05.230049 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.229976 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-cluster-tls-config\") pod \"8d9132c3-a362-42c1-a5fd-189ef1277034\" (UID: \"8d9132c3-a362-42c1-a5fd-189ef1277034\") " Jul 17 17:52:05.230049 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.229999 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9132c3-a362-42c1-a5fd-189ef1277034-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "8d9132c3-a362-42c1-a5fd-189ef1277034" (UID: "8d9132c3-a362-42c1-a5fd-189ef1277034"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:52:05.237464 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.230233 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9132c3-a362-42c1-a5fd-189ef1277034-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "8d9132c3-a362-42c1-a5fd-189ef1277034" (UID: "8d9132c3-a362-42c1-a5fd-189ef1277034"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:52:05.237464 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.230254 2574 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8d9132c3-a362-42c1-a5fd-189ef1277034-alertmanager-main-db\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:05.237464 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.230715 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9132c3-a362-42c1-a5fd-189ef1277034-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "8d9132c3-a362-42c1-a5fd-189ef1277034" (UID: "8d9132c3-a362-42c1-a5fd-189ef1277034"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:52:05.237464 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.235319 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "8d9132c3-a362-42c1-a5fd-189ef1277034" (UID: "8d9132c3-a362-42c1-a5fd-189ef1277034"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:52:05.237464 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.236388 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "8d9132c3-a362-42c1-a5fd-189ef1277034" (UID: "8d9132c3-a362-42c1-a5fd-189ef1277034"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:52:05.237464 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.236934 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9132c3-a362-42c1-a5fd-189ef1277034-kube-api-access-h4hrh" (OuterVolumeSpecName: "kube-api-access-h4hrh") pod "8d9132c3-a362-42c1-a5fd-189ef1277034" (UID: "8d9132c3-a362-42c1-a5fd-189ef1277034"). InnerVolumeSpecName "kube-api-access-h4hrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:52:05.238693 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.238663 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "8d9132c3-a362-42c1-a5fd-189ef1277034" (UID: "8d9132c3-a362-42c1-a5fd-189ef1277034"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:52:05.238847 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.238687 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "8d9132c3-a362-42c1-a5fd-189ef1277034" (UID: "8d9132c3-a362-42c1-a5fd-189ef1277034"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:52:05.238847 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.238757 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-config-volume" (OuterVolumeSpecName: "config-volume") pod "8d9132c3-a362-42c1-a5fd-189ef1277034" (UID: "8d9132c3-a362-42c1-a5fd-189ef1277034"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:52:05.239424 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.239402 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9132c3-a362-42c1-a5fd-189ef1277034-config-out" (OuterVolumeSpecName: "config-out") pod "8d9132c3-a362-42c1-a5fd-189ef1277034" (UID: "8d9132c3-a362-42c1-a5fd-189ef1277034"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:52:05.239536 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.239508 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9132c3-a362-42c1-a5fd-189ef1277034-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8d9132c3-a362-42c1-a5fd-189ef1277034" (UID: "8d9132c3-a362-42c1-a5fd-189ef1277034"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:52:05.244798 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.244752 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "8d9132c3-a362-42c1-a5fd-189ef1277034" (UID: "8d9132c3-a362-42c1-a5fd-189ef1277034"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:52:05.254653 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.254618 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-web-config" (OuterVolumeSpecName: "web-config") pod "8d9132c3-a362-42c1-a5fd-189ef1277034" (UID: "8d9132c3-a362-42c1-a5fd-189ef1277034"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:52:05.331375 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.331299 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:05.331375 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.331344 2574 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d9132c3-a362-42c1-a5fd-189ef1277034-config-out\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:05.331375 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.331381 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:05.331375 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.331392 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:05.331662 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.331402 2574 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d9132c3-a362-42c1-a5fd-189ef1277034-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:05.331662 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.331410 2574 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-config-volume\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:05.331662 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.331419 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4hrh\" (UniqueName: \"kubernetes.io/projected/8d9132c3-a362-42c1-a5fd-189ef1277034-kube-api-access-h4hrh\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:05.331662 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.331428 2574 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d9132c3-a362-42c1-a5fd-189ef1277034-tls-assets\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:05.331662 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.331436 2574 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-web-config\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:05.331662 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.331443 2574 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-cluster-tls-config\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:05.331662 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.331451 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8d9132c3-a362-42c1-a5fd-189ef1277034-secret-alertmanager-main-tls\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:05.331662 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.331459 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d9132c3-a362-42c1-a5fd-189ef1277034-metrics-client-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:05.893302 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.893265 2574 generic.go:356] "Generic (PLEG): container finished" podID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerID="67c94ef00288b7720b677f562b743ef5e2dc705d3a5782981aaabb8b3b313c59" exitCode=0 Jul 17 17:52:05.893513 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.893363 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:05.893513 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.893347 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d9132c3-a362-42c1-a5fd-189ef1277034","Type":"ContainerDied","Data":"67c94ef00288b7720b677f562b743ef5e2dc705d3a5782981aaabb8b3b313c59"} Jul 17 17:52:05.893513 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.893468 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d9132c3-a362-42c1-a5fd-189ef1277034","Type":"ContainerDied","Data":"3e6f10c3257cac318726c587cb60d1012980295a5873063a6b9830621f43ae94"} Jul 17 17:52:05.893513 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.893504 2574 scope.go:117] "RemoveContainer" containerID="d12b62d83719f065cefa7507464d95b2b02f41633d20fddefc9f49d2abb71d88" Jul 17 17:52:05.902922 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.902906 2574 scope.go:117] "RemoveContainer" containerID="d3a4afa436b78a87db2e95f567edd649283955f9e8e93d1a43a18f2520d90fc7" Jul 17 17:52:05.910679 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.910663 2574 scope.go:117] "RemoveContainer" containerID="9882987cca4122f37c4ecdd490dc7a0fc87467f981f2b0e072182641401d134b" Jul 17 17:52:05.917999 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.917973 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:52:05.919066 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.918924 2574 scope.go:117] "RemoveContainer" containerID="67c94ef00288b7720b677f562b743ef5e2dc705d3a5782981aaabb8b3b313c59" Jul 17 17:52:05.921784 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.921761 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:52:05.927037 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.927018 2574 scope.go:117] "RemoveContainer" containerID="fc59f95099605225b3aa67e76e3bb6fe8395f8db48f22428a8ede8f0a703833c" Jul 17 17:52:05.934511 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.934490 2574 scope.go:117] "RemoveContainer" containerID="525257b95d1069d04b432583562598f0cc005d2eb8b7e4676703957a3aa54dd0" Jul 17 17:52:05.942542 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.942523 2574 scope.go:117] "RemoveContainer" containerID="b74c1974a28db1ea9df66c6c8f8701455780ad3d953a009c09ceb2e7b8154aa7" Jul 17 17:52:05.947320 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947297 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:52:05.947664 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947650 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73507cce-1862-4050-a43c-d568330c5c59" containerName="console" Jul 17 17:52:05.947709 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947667 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="73507cce-1862-4050-a43c-d568330c5c59" containerName="console" Jul 17 17:52:05.947709 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947674 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="alertmanager" Jul 17 17:52:05.947709 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947680 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="alertmanager" Jul 17 17:52:05.947709 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947688 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="kube-rbac-proxy-web" Jul 17 17:52:05.947709 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947694 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="kube-rbac-proxy-web" Jul 17 17:52:05.947709 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947704 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="kube-rbac-proxy" Jul 17 17:52:05.947709 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947709 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="kube-rbac-proxy" Jul 17 17:52:05.948029 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947721 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="init-config-reloader" Jul 17 17:52:05.948029 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947726 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="init-config-reloader" Jul 17 17:52:05.948029 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947735 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="prom-label-proxy" Jul 17 17:52:05.948029 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947740 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="prom-label-proxy" Jul 17 17:52:05.948029 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947748 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="config-reloader" Jul 17 17:52:05.948029 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947753 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="config-reloader" Jul 17 17:52:05.948029 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947759 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="kube-rbac-proxy-metric" Jul 17 17:52:05.948029 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947764 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="kube-rbac-proxy-metric" Jul 17 17:52:05.948029 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947815 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="alertmanager" Jul 17 17:52:05.948029 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947823 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="73507cce-1862-4050-a43c-d568330c5c59" containerName="console" Jul 17 17:52:05.948029 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947830 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="config-reloader" Jul 17 17:52:05.948029 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947835 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="kube-rbac-proxy" Jul 17 17:52:05.948029 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947841 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="kube-rbac-proxy-metric" Jul 17 17:52:05.948029 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947849 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="kube-rbac-proxy-web" Jul 17 17:52:05.948029 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.947854 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" containerName="prom-label-proxy" Jul 17 17:52:05.951521 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.951499 2574 scope.go:117] "RemoveContainer" containerID="d12b62d83719f065cefa7507464d95b2b02f41633d20fddefc9f49d2abb71d88" Jul 17 17:52:05.951857 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:52:05.951829 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12b62d83719f065cefa7507464d95b2b02f41633d20fddefc9f49d2abb71d88\": container with ID starting with d12b62d83719f065cefa7507464d95b2b02f41633d20fddefc9f49d2abb71d88 not found: ID does not exist" containerID="d12b62d83719f065cefa7507464d95b2b02f41633d20fddefc9f49d2abb71d88" Jul 17 17:52:05.951968 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.951865 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12b62d83719f065cefa7507464d95b2b02f41633d20fddefc9f49d2abb71d88"} err="failed to get container status \"d12b62d83719f065cefa7507464d95b2b02f41633d20fddefc9f49d2abb71d88\": rpc error: code = NotFound desc = could not find container \"d12b62d83719f065cefa7507464d95b2b02f41633d20fddefc9f49d2abb71d88\": container with ID starting with d12b62d83719f065cefa7507464d95b2b02f41633d20fddefc9f49d2abb71d88 not found: ID does not exist" Jul 17 17:52:05.951968 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.951891 2574 scope.go:117] "RemoveContainer" containerID="d3a4afa436b78a87db2e95f567edd649283955f9e8e93d1a43a18f2520d90fc7" Jul 17 17:52:05.952199 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:52:05.952176 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3a4afa436b78a87db2e95f567edd649283955f9e8e93d1a43a18f2520d90fc7\": container with ID starting with d3a4afa436b78a87db2e95f567edd649283955f9e8e93d1a43a18f2520d90fc7 not found: ID does not exist" containerID="d3a4afa436b78a87db2e95f567edd649283955f9e8e93d1a43a18f2520d90fc7" Jul 17 17:52:05.952249 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.952207 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3a4afa436b78a87db2e95f567edd649283955f9e8e93d1a43a18f2520d90fc7"} err="failed to get container status \"d3a4afa436b78a87db2e95f567edd649283955f9e8e93d1a43a18f2520d90fc7\": rpc error: code = NotFound desc = could not find container \"d3a4afa436b78a87db2e95f567edd649283955f9e8e93d1a43a18f2520d90fc7\": container with ID starting with d3a4afa436b78a87db2e95f567edd649283955f9e8e93d1a43a18f2520d90fc7 not found: ID does not exist" Jul 17 17:52:05.952249 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.952228 2574 scope.go:117] "RemoveContainer" containerID="9882987cca4122f37c4ecdd490dc7a0fc87467f981f2b0e072182641401d134b" Jul 17 17:52:05.952571 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:52:05.952553 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9882987cca4122f37c4ecdd490dc7a0fc87467f981f2b0e072182641401d134b\": container with ID starting with 9882987cca4122f37c4ecdd490dc7a0fc87467f981f2b0e072182641401d134b not found: ID does not exist" containerID="9882987cca4122f37c4ecdd490dc7a0fc87467f981f2b0e072182641401d134b" Jul 17 17:52:05.952637 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.952574 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9882987cca4122f37c4ecdd490dc7a0fc87467f981f2b0e072182641401d134b"} err="failed to get container status \"9882987cca4122f37c4ecdd490dc7a0fc87467f981f2b0e072182641401d134b\": rpc error: code = NotFound desc = could not find container \"9882987cca4122f37c4ecdd490dc7a0fc87467f981f2b0e072182641401d134b\": container with ID starting with 9882987cca4122f37c4ecdd490dc7a0fc87467f981f2b0e072182641401d134b not found: ID does not exist" Jul 17 17:52:05.952637 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.952588 2574 scope.go:117] "RemoveContainer" containerID="67c94ef00288b7720b677f562b743ef5e2dc705d3a5782981aaabb8b3b313c59" Jul 17 17:52:05.952806 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:52:05.952792 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67c94ef00288b7720b677f562b743ef5e2dc705d3a5782981aaabb8b3b313c59\": container with ID starting with 67c94ef00288b7720b677f562b743ef5e2dc705d3a5782981aaabb8b3b313c59 not found: ID does not exist" containerID="67c94ef00288b7720b677f562b743ef5e2dc705d3a5782981aaabb8b3b313c59" Jul 17 17:52:05.952844 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.952812 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67c94ef00288b7720b677f562b743ef5e2dc705d3a5782981aaabb8b3b313c59"} err="failed to get container status \"67c94ef00288b7720b677f562b743ef5e2dc705d3a5782981aaabb8b3b313c59\": rpc error: code = NotFound desc = could not find container \"67c94ef00288b7720b677f562b743ef5e2dc705d3a5782981aaabb8b3b313c59\": container with ID starting with 67c94ef00288b7720b677f562b743ef5e2dc705d3a5782981aaabb8b3b313c59 not found: ID does not exist" Jul 17 17:52:05.952844 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.952830 2574 scope.go:117] "RemoveContainer" containerID="fc59f95099605225b3aa67e76e3bb6fe8395f8db48f22428a8ede8f0a703833c" Jul 17 17:52:05.953044 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:52:05.953027 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc59f95099605225b3aa67e76e3bb6fe8395f8db48f22428a8ede8f0a703833c\": container with ID starting with fc59f95099605225b3aa67e76e3bb6fe8395f8db48f22428a8ede8f0a703833c not found: ID does not exist" containerID="fc59f95099605225b3aa67e76e3bb6fe8395f8db48f22428a8ede8f0a703833c" Jul 17 17:52:05.953088 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.953052 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc59f95099605225b3aa67e76e3bb6fe8395f8db48f22428a8ede8f0a703833c"} err="failed to get container status \"fc59f95099605225b3aa67e76e3bb6fe8395f8db48f22428a8ede8f0a703833c\": rpc error: code = NotFound desc = could not find container \"fc59f95099605225b3aa67e76e3bb6fe8395f8db48f22428a8ede8f0a703833c\": container with ID starting with fc59f95099605225b3aa67e76e3bb6fe8395f8db48f22428a8ede8f0a703833c not found: ID does not exist" Jul 17 17:52:05.953124 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.953091 2574 scope.go:117] "RemoveContainer" containerID="525257b95d1069d04b432583562598f0cc005d2eb8b7e4676703957a3aa54dd0" Jul 17 17:52:05.953397 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:52:05.953379 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"525257b95d1069d04b432583562598f0cc005d2eb8b7e4676703957a3aa54dd0\": container with ID starting with 525257b95d1069d04b432583562598f0cc005d2eb8b7e4676703957a3aa54dd0 not found: ID does not exist" containerID="525257b95d1069d04b432583562598f0cc005d2eb8b7e4676703957a3aa54dd0" Jul 17 17:52:05.953453 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.953404 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525257b95d1069d04b432583562598f0cc005d2eb8b7e4676703957a3aa54dd0"} err="failed to get container status \"525257b95d1069d04b432583562598f0cc005d2eb8b7e4676703957a3aa54dd0\": rpc error: code = NotFound desc = could not find container \"525257b95d1069d04b432583562598f0cc005d2eb8b7e4676703957a3aa54dd0\": container with ID starting with 525257b95d1069d04b432583562598f0cc005d2eb8b7e4676703957a3aa54dd0 not found: ID does not exist" Jul 17 17:52:05.953453 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.953419 2574 scope.go:117] "RemoveContainer" containerID="b74c1974a28db1ea9df66c6c8f8701455780ad3d953a009c09ceb2e7b8154aa7" Jul 17 17:52:05.953714 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:52:05.953692 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b74c1974a28db1ea9df66c6c8f8701455780ad3d953a009c09ceb2e7b8154aa7\": container with ID starting with b74c1974a28db1ea9df66c6c8f8701455780ad3d953a009c09ceb2e7b8154aa7 not found: ID does not exist" containerID="b74c1974a28db1ea9df66c6c8f8701455780ad3d953a009c09ceb2e7b8154aa7" Jul 17 17:52:05.953772 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.953723 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b74c1974a28db1ea9df66c6c8f8701455780ad3d953a009c09ceb2e7b8154aa7"} err="failed to get container status \"b74c1974a28db1ea9df66c6c8f8701455780ad3d953a009c09ceb2e7b8154aa7\": rpc error: code = NotFound desc = could not find container \"b74c1974a28db1ea9df66c6c8f8701455780ad3d953a009c09ceb2e7b8154aa7\": container with ID starting with b74c1974a28db1ea9df66c6c8f8701455780ad3d953a009c09ceb2e7b8154aa7 not found: ID does not exist" Jul 17 17:52:05.954555 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.954539 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:05.957394 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.957373 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 17 17:52:05.957514 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.957417 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 17 17:52:05.957655 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.957638 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 17 17:52:05.957739 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.957674 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 17 17:52:05.957800 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.957741 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 17 17:52:05.957800 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.957769 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 17 17:52:05.957905 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.957818 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 17 17:52:05.958094 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.958077 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-npbf6\"" Jul 17 17:52:05.958166 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.958124 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 17 17:52:05.962833 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.962795 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 17 17:52:05.965833 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:05.965809 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:52:06.037485 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.037452 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg8t5\" (UniqueName: \"kubernetes.io/projected/c0ab0b93-d246-49b1-bddd-f1bcba272d12-kube-api-access-dg8t5\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.037485 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.037494 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ab0b93-d246-49b1-bddd-f1bcba272d12-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.037685 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.037529 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0ab0b93-d246-49b1-bddd-f1bcba272d12-config-volume\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.037685 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.037560 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0ab0b93-d246-49b1-bddd-f1bcba272d12-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.037685 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.037601 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0ab0b93-d246-49b1-bddd-f1bcba272d12-config-out\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.037685 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.037626 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0ab0b93-d246-49b1-bddd-f1bcba272d12-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.037685 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.037646 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0ab0b93-d246-49b1-bddd-f1bcba272d12-web-config\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.037685 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.037674 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0ab0b93-d246-49b1-bddd-f1bcba272d12-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.037854 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.037731 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0ab0b93-d246-49b1-bddd-f1bcba272d12-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.037854 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.037762 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0ab0b93-d246-49b1-bddd-f1bcba272d12-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.037854 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.037788 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0ab0b93-d246-49b1-bddd-f1bcba272d12-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.037854 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.037803 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0ab0b93-d246-49b1-bddd-f1bcba272d12-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.037854 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.037827 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0ab0b93-d246-49b1-bddd-f1bcba272d12-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.138935 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.138893 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0ab0b93-d246-49b1-bddd-f1bcba272d12-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.138935 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.138937 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0ab0b93-d246-49b1-bddd-f1bcba272d12-config-out\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.139178 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.138962 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0ab0b93-d246-49b1-bddd-f1bcba272d12-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.139178 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.138979 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0ab0b93-d246-49b1-bddd-f1bcba272d12-web-config\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.139178 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.139006 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0ab0b93-d246-49b1-bddd-f1bcba272d12-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.139178 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.139031 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0ab0b93-d246-49b1-bddd-f1bcba272d12-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.139178 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.139060 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0ab0b93-d246-49b1-bddd-f1bcba272d12-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.139178 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.139094 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0ab0b93-d246-49b1-bddd-f1bcba272d12-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.139178 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.139118 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0ab0b93-d246-49b1-bddd-f1bcba272d12-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.139178 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.139151 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0ab0b93-d246-49b1-bddd-f1bcba272d12-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.139178 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.139179 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg8t5\" (UniqueName: \"kubernetes.io/projected/c0ab0b93-d246-49b1-bddd-f1bcba272d12-kube-api-access-dg8t5\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.139617 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.139201 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ab0b93-d246-49b1-bddd-f1bcba272d12-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.139617 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.139249 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0ab0b93-d246-49b1-bddd-f1bcba272d12-config-volume\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.139617 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.139402 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0ab0b93-d246-49b1-bddd-f1bcba272d12-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.142471 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.142306 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0ab0b93-d246-49b1-bddd-f1bcba272d12-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.142471 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.142399 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0ab0b93-d246-49b1-bddd-f1bcba272d12-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.142471 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.142399 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0ab0b93-d246-49b1-bddd-f1bcba272d12-config-volume\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.142820 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.142481 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0ab0b93-d246-49b1-bddd-f1bcba272d12-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.142820 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.142515 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0ab0b93-d246-49b1-bddd-f1bcba272d12-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.142820 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.142550 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0ab0b93-d246-49b1-bddd-f1bcba272d12-config-out\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.142820 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.142567 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0ab0b93-d246-49b1-bddd-f1bcba272d12-web-config\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.143101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.143078 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0ab0b93-d246-49b1-bddd-f1bcba272d12-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.143274 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.143256 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ab0b93-d246-49b1-bddd-f1bcba272d12-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.143605 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.143556 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0ab0b93-d246-49b1-bddd-f1bcba272d12-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.144617 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.144596 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0ab0b93-d246-49b1-bddd-f1bcba272d12-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.151317 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.151293 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg8t5\" (UniqueName: \"kubernetes.io/projected/c0ab0b93-d246-49b1-bddd-f1bcba272d12-kube-api-access-dg8t5\") pod \"alertmanager-main-0\" (UID: \"c0ab0b93-d246-49b1-bddd-f1bcba272d12\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.266039 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.266001 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 17:52:06.413718 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.413636 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 17:52:06.417208 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:52:06.417175 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ab0b93_d246_49b1_bddd_f1bcba272d12.slice/crio-be09f28af29b36706a4a3b23168164cd30268049a611656ffe41de5a95f84cf4 WatchSource:0}: Error finding container be09f28af29b36706a4a3b23168164cd30268049a611656ffe41de5a95f84cf4: Status 404 returned error can't find the container with id be09f28af29b36706a4a3b23168164cd30268049a611656ffe41de5a95f84cf4 Jul 17 17:52:06.899029 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.898994 2574 generic.go:356] "Generic (PLEG): container finished" podID="c0ab0b93-d246-49b1-bddd-f1bcba272d12" containerID="a2afd9364f29f9b840b06ac64a48aa4298f516f7d227d2ce4a53088f6f7ece95" exitCode=0 Jul 17 17:52:06.899240 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.899081 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0ab0b93-d246-49b1-bddd-f1bcba272d12","Type":"ContainerDied","Data":"a2afd9364f29f9b840b06ac64a48aa4298f516f7d227d2ce4a53088f6f7ece95"} Jul 17 17:52:06.899240 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.899155 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0ab0b93-d246-49b1-bddd-f1bcba272d12","Type":"ContainerStarted","Data":"be09f28af29b36706a4a3b23168164cd30268049a611656ffe41de5a95f84cf4"} Jul 17 17:52:06.899892 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.899830 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:52:06.965064 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.965027 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:52:06.989587 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:06.989555 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d9132c3-a362-42c1-a5fd-189ef1277034" path="/var/lib/kubelet/pods/8d9132c3-a362-42c1-a5fd-189ef1277034/volumes" Jul 17 17:52:07.910712 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:07.910679 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0ab0b93-d246-49b1-bddd-f1bcba272d12","Type":"ContainerStarted","Data":"8fd530c5911b0027190639725a04f28dc8d6957ddb6362c588142adecc174c4a"} Jul 17 17:52:07.911120 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:07.910747 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0ab0b93-d246-49b1-bddd-f1bcba272d12","Type":"ContainerStarted","Data":"3cc75a1da687c146c3e22f10b71d6ce4302baa26d3194cd5b282dbebb1df88ed"} Jul 17 17:52:07.911120 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:07.910776 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0ab0b93-d246-49b1-bddd-f1bcba272d12","Type":"ContainerStarted","Data":"4e0aa74e9a19df2a7c054fb74364973eb4356fc38c6b8457f475bc3a4e4a31cc"} Jul 17 17:52:07.911120 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:07.910803 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0ab0b93-d246-49b1-bddd-f1bcba272d12","Type":"ContainerStarted","Data":"7e17d4b322397ada8d8ef524b2677e8470ad8cda4be2b8597cec696847954d99"} Jul 17 17:52:07.911120 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:07.910827 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0ab0b93-d246-49b1-bddd-f1bcba272d12","Type":"ContainerStarted","Data":"63b6aa0223606a83875e020a8bac8cb388d64b07ce14a93f4b46b82c9dec3e1c"} Jul 17 17:52:07.911120 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:07.910848 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0ab0b93-d246-49b1-bddd-f1bcba272d12","Type":"ContainerStarted","Data":"6c0c1334e274172d13b371566224358554593035f454b4fec05be6a2a9cbbd7c"} Jul 17 17:52:07.936921 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:07.936856 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.9200271190000002 podStartE2EDuration="2.936835522s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.808403ms" isStatefulPod=false podCreationTimestamp="2026-07-17 17:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:52:07.934798374 +0000 UTC m=+139.522751065" watchObservedRunningTime="2026-07-17 17:52:07.936835522 +0000 UTC m=+139.524788214" Jul 17 17:52:08.168886 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.168798 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:52:08.169571 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.169178 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="kube-rbac-proxy" containerID="cri-o://dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699" gracePeriod=600 Jul 17 17:52:08.169571 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.169344 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="kube-rbac-proxy-web" containerID="cri-o://6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031" gracePeriod=600 Jul 17 17:52:08.169571 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.169425 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="kube-rbac-proxy-thanos" containerID="cri-o://c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25" gracePeriod=600 Jul 17 17:52:08.169571 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.169494 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="thanos-sidecar" containerID="cri-o://f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488" gracePeriod=600 Jul 17 17:52:08.169571 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.169166 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="prometheus" containerID="cri-o://4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7" gracePeriod=600 Jul 17 17:52:08.169897 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.169554 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="config-reloader" containerID="cri-o://5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20" gracePeriod=600 Jul 17 17:52:08.427250 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.427179 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:08.463587 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.463210 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b978bdb1-82ea-40a0-b843-a148a14e5c22-config-out\") pod \"b978bdb1-82ea-40a0-b843-a148a14e5c22\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " Jul 17 17:52:08.463587 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.463264 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgcfh\" (UniqueName: \"kubernetes.io/projected/b978bdb1-82ea-40a0-b843-a148a14e5c22-kube-api-access-vgcfh\") pod \"b978bdb1-82ea-40a0-b843-a148a14e5c22\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " Jul 17 17:52:08.463587 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.463299 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-web-config\") pod \"b978bdb1-82ea-40a0-b843-a148a14e5c22\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " Jul 17 17:52:08.463587 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.463326 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-configmap-metrics-client-ca\") pod \"b978bdb1-82ea-40a0-b843-a148a14e5c22\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " Jul 17 17:52:08.463587 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.463388 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b978bdb1-82ea-40a0-b843-a148a14e5c22\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " Jul 17 17:52:08.463587 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.463427 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-prometheus-k8s-tls\") pod \"b978bdb1-82ea-40a0-b843-a148a14e5c22\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " Jul 17 17:52:08.463587 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.463486 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-metrics-client-certs\") pod \"b978bdb1-82ea-40a0-b843-a148a14e5c22\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " Jul 17 17:52:08.463587 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.463508 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-rulefiles-1\") pod \"b978bdb1-82ea-40a0-b843-a148a14e5c22\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " Jul 17 17:52:08.463587 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.463531 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-thanos-prometheus-http-client-file\") pod \"b978bdb1-82ea-40a0-b843-a148a14e5c22\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " Jul 17 17:52:08.463587 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.463577 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-configmap-kubelet-serving-ca-bundle\") pod \"b978bdb1-82ea-40a0-b843-a148a14e5c22\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " Jul 17 17:52:08.463587 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.463600 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b978bdb1-82ea-40a0-b843-a148a14e5c22\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " Jul 17 17:52:08.464230 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.463627 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-kube-rbac-proxy\") pod \"b978bdb1-82ea-40a0-b843-a148a14e5c22\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " Jul 17 17:52:08.464230 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.463652 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-grpc-tls\") pod \"b978bdb1-82ea-40a0-b843-a148a14e5c22\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " Jul 17 17:52:08.464230 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.463677 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-trusted-ca-bundle\") pod \"b978bdb1-82ea-40a0-b843-a148a14e5c22\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " Jul 17 17:52:08.464230 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.463724 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-rulefiles-2\") pod \"b978bdb1-82ea-40a0-b843-a148a14e5c22\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " Jul 17 17:52:08.464230 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.463749 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-config\") pod \"b978bdb1-82ea-40a0-b843-a148a14e5c22\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " Jul 17 17:52:08.464230 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.463797 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-db\") pod \"b978bdb1-82ea-40a0-b843-a148a14e5c22\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " Jul 17 17:52:08.464230 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.463823 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-configmap-serving-certs-ca-bundle\") pod \"b978bdb1-82ea-40a0-b843-a148a14e5c22\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " Jul 17 17:52:08.464230 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.463849 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-rulefiles-0\") pod \"b978bdb1-82ea-40a0-b843-a148a14e5c22\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " Jul 17 17:52:08.464230 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.463882 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b978bdb1-82ea-40a0-b843-a148a14e5c22-tls-assets\") pod \"b978bdb1-82ea-40a0-b843-a148a14e5c22\" (UID: \"b978bdb1-82ea-40a0-b843-a148a14e5c22\") " Jul 17 17:52:08.465379 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.465268 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b978bdb1-82ea-40a0-b843-a148a14e5c22" (UID: "b978bdb1-82ea-40a0-b843-a148a14e5c22"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:52:08.465759 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.465695 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "b978bdb1-82ea-40a0-b843-a148a14e5c22" (UID: "b978bdb1-82ea-40a0-b843-a148a14e5c22"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:52:08.467155 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.467122 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b978bdb1-82ea-40a0-b843-a148a14e5c22" (UID: "b978bdb1-82ea-40a0-b843-a148a14e5c22"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:52:08.467460 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.467433 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b978bdb1-82ea-40a0-b843-a148a14e5c22" (UID: "b978bdb1-82ea-40a0-b843-a148a14e5c22"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:52:08.468006 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.467954 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "b978bdb1-82ea-40a0-b843-a148a14e5c22" (UID: "b978bdb1-82ea-40a0-b843-a148a14e5c22"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:52:08.468006 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.467971 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b978bdb1-82ea-40a0-b843-a148a14e5c22" (UID: "b978bdb1-82ea-40a0-b843-a148a14e5c22"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:52:08.468971 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.468915 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b978bdb1-82ea-40a0-b843-a148a14e5c22" (UID: "b978bdb1-82ea-40a0-b843-a148a14e5c22"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:52:08.469139 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.469112 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b978bdb1-82ea-40a0-b843-a148a14e5c22" (UID: "b978bdb1-82ea-40a0-b843-a148a14e5c22"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 17:52:08.472371 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.472289 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b978bdb1-82ea-40a0-b843-a148a14e5c22-kube-api-access-vgcfh" (OuterVolumeSpecName: "kube-api-access-vgcfh") pod "b978bdb1-82ea-40a0-b843-a148a14e5c22" (UID: "b978bdb1-82ea-40a0-b843-a148a14e5c22"). InnerVolumeSpecName "kube-api-access-vgcfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:52:08.472371 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.472301 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b978bdb1-82ea-40a0-b843-a148a14e5c22" (UID: "b978bdb1-82ea-40a0-b843-a148a14e5c22"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:52:08.472547 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.472442 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b978bdb1-82ea-40a0-b843-a148a14e5c22" (UID: "b978bdb1-82ea-40a0-b843-a148a14e5c22"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:52:08.472627 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.472597 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b978bdb1-82ea-40a0-b843-a148a14e5c22" (UID: "b978bdb1-82ea-40a0-b843-a148a14e5c22"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:52:08.473063 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.473011 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b978bdb1-82ea-40a0-b843-a148a14e5c22-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b978bdb1-82ea-40a0-b843-a148a14e5c22" (UID: "b978bdb1-82ea-40a0-b843-a148a14e5c22"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:52:08.473278 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.473252 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b978bdb1-82ea-40a0-b843-a148a14e5c22" (UID: "b978bdb1-82ea-40a0-b843-a148a14e5c22"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:52:08.473707 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.473665 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-config" (OuterVolumeSpecName: "config") pod "b978bdb1-82ea-40a0-b843-a148a14e5c22" (UID: "b978bdb1-82ea-40a0-b843-a148a14e5c22"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:52:08.474170 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.474120 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b978bdb1-82ea-40a0-b843-a148a14e5c22" (UID: "b978bdb1-82ea-40a0-b843-a148a14e5c22"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:52:08.474679 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.474654 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b978bdb1-82ea-40a0-b843-a148a14e5c22" (UID: "b978bdb1-82ea-40a0-b843-a148a14e5c22"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:52:08.477153 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.477101 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b978bdb1-82ea-40a0-b843-a148a14e5c22" (UID: "b978bdb1-82ea-40a0-b843-a148a14e5c22"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:52:08.477595 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.477561 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b978bdb1-82ea-40a0-b843-a148a14e5c22-config-out" (OuterVolumeSpecName: "config-out") pod "b978bdb1-82ea-40a0-b843-a148a14e5c22" (UID: "b978bdb1-82ea-40a0-b843-a148a14e5c22"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:52:08.486294 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.486250 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-web-config" (OuterVolumeSpecName: "web-config") pod "b978bdb1-82ea-40a0-b843-a148a14e5c22" (UID: "b978bdb1-82ea-40a0-b843-a148a14e5c22"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 17:52:08.564609 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.564571 2574 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:08.564609 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.564602 2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:08.564609 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.564615 2574 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-kube-rbac-proxy\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:08.564609 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.564624 2574 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-grpc-tls\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:08.564888 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.564632 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:08.564888 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.564643 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:08.564888 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.564651 2574 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-config\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:08.564888 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.564659 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-db\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:08.564888 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.564669 2574 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:08.564888 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.564677 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:08.564888 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.564686 2574 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b978bdb1-82ea-40a0-b843-a148a14e5c22-tls-assets\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:08.564888 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.564695 2574 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b978bdb1-82ea-40a0-b843-a148a14e5c22-config-out\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:08.564888 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.564702 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgcfh\" (UniqueName: \"kubernetes.io/projected/b978bdb1-82ea-40a0-b843-a148a14e5c22-kube-api-access-vgcfh\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:08.564888 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.564710 2574 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-web-config\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:08.564888 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.564720 2574 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-configmap-metrics-client-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:08.564888 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.564728 2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:08.564888 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.564737 2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:08.564888 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.564746 2574 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-secret-metrics-client-certs\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:08.564888 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.564756 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b978bdb1-82ea-40a0-b843-a148a14e5c22-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:08.564888 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.564764 2574 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b978bdb1-82ea-40a0-b843-a148a14e5c22-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:52:08.919169 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.919132 2574 generic.go:356] "Generic (PLEG): container finished" podID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerID="c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25" exitCode=0 Jul 17 17:52:08.919169 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.919160 2574 generic.go:356] "Generic (PLEG): container finished" podID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerID="dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699" exitCode=0 Jul 17 17:52:08.919169 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.919167 2574 generic.go:356] "Generic (PLEG): container finished" podID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerID="6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031" exitCode=0 Jul 17 17:52:08.919169 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.919172 2574 generic.go:356] "Generic (PLEG): container finished" podID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerID="f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488" exitCode=0 Jul 17 17:52:08.919169 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.919177 2574 generic.go:356] "Generic (PLEG): container finished" podID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerID="5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20" exitCode=0 Jul 17 17:52:08.919822 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.919182 2574 generic.go:356] "Generic (PLEG): container finished" podID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerID="4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7" exitCode=0 Jul 17 17:52:08.919822 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.919210 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:08.919822 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.919209 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b978bdb1-82ea-40a0-b843-a148a14e5c22","Type":"ContainerDied","Data":"c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25"} Jul 17 17:52:08.919822 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.919281 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b978bdb1-82ea-40a0-b843-a148a14e5c22","Type":"ContainerDied","Data":"dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699"} Jul 17 17:52:08.919822 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.919321 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b978bdb1-82ea-40a0-b843-a148a14e5c22","Type":"ContainerDied","Data":"6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031"} Jul 17 17:52:08.919822 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.919348 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b978bdb1-82ea-40a0-b843-a148a14e5c22","Type":"ContainerDied","Data":"f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488"} Jul 17 17:52:08.919822 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.919382 2574 scope.go:117] "RemoveContainer" containerID="c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25" Jul 17 17:52:08.919822 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.919404 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b978bdb1-82ea-40a0-b843-a148a14e5c22","Type":"ContainerDied","Data":"5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20"} Jul 17 17:52:08.919822 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.919430 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b978bdb1-82ea-40a0-b843-a148a14e5c22","Type":"ContainerDied","Data":"4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7"} Jul 17 17:52:08.919822 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.919453 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b978bdb1-82ea-40a0-b843-a148a14e5c22","Type":"ContainerDied","Data":"edd1dc4ebac9094aa53f385001c5a9018ccd476789473becac2602cdf72df0b0"} Jul 17 17:52:08.927935 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.927918 2574 scope.go:117] "RemoveContainer" containerID="dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699" Jul 17 17:52:08.935260 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.935242 2574 scope.go:117] "RemoveContainer" containerID="6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031" Jul 17 17:52:08.942653 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.942627 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:52:08.943052 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.943035 2574 scope.go:117] "RemoveContainer" containerID="f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488" Jul 17 17:52:08.948816 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.948794 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:52:08.951181 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.951158 2574 scope.go:117] "RemoveContainer" containerID="5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20" Jul 17 17:52:08.966863 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.966826 2574 scope.go:117] "RemoveContainer" containerID="4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7" Jul 17 17:52:08.973050 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.973025 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:52:08.973500 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.973484 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="kube-rbac-proxy" Jul 17 17:52:08.973565 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.973503 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="kube-rbac-proxy" Jul 17 17:52:08.973565 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.973525 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="config-reloader" Jul 17 17:52:08.973565 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.973533 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="config-reloader" Jul 17 17:52:08.973565 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.973540 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="thanos-sidecar" Jul 17 17:52:08.973565 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.973545 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="thanos-sidecar" Jul 17 17:52:08.973565 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.973554 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="prometheus" Jul 17 17:52:08.973565 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.973560 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="prometheus" Jul 17 17:52:08.973836 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.973574 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="init-config-reloader" Jul 17 17:52:08.973836 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.973582 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="init-config-reloader" Jul 17 17:52:08.973836 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.973591 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="kube-rbac-proxy-web" Jul 17 17:52:08.973836 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.973596 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="kube-rbac-proxy-web" Jul 17 17:52:08.973836 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.973605 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="kube-rbac-proxy-thanos" Jul 17 17:52:08.973836 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.973610 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="kube-rbac-proxy-thanos" Jul 17 17:52:08.973836 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.973670 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="thanos-sidecar" Jul 17 17:52:08.973836 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.973682 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="kube-rbac-proxy" Jul 17 17:52:08.973836 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.973691 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="kube-rbac-proxy-web" Jul 17 17:52:08.973836 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.973699 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="kube-rbac-proxy-thanos" Jul 17 17:52:08.973836 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.973706 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="config-reloader" Jul 17 17:52:08.973836 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.973717 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" containerName="prometheus" Jul 17 17:52:08.977084 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.977067 2574 scope.go:117] "RemoveContainer" containerID="f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42" Jul 17 17:52:08.981080 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.981055 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:08.984130 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.984107 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-n4gp9\"" Jul 17 17:52:08.984230 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.984171 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 17 17:52:08.984230 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.984107 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 17 17:52:08.984415 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.984391 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 17 17:52:08.984591 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.984534 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 17 17:52:08.984591 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.984560 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-b61sba05c5kus\"" Jul 17 17:52:08.984712 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.984609 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 17 17:52:08.984712 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.984643 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 17 17:52:08.985015 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.984989 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 17 17:52:08.985100 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.985059 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 17 17:52:08.985183 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.985167 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 17 17:52:08.985772 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.985575 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 17 17:52:08.985864 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.985839 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 17 17:52:08.985975 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.985960 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 17 17:52:08.987211 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.987189 2574 scope.go:117] "RemoveContainer" containerID="c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25" Jul 17 17:52:08.987720 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:52:08.987685 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25\": container with ID starting with c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25 not found: ID does not exist" containerID="c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25" Jul 17 17:52:08.987820 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.987733 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25"} err="failed to get container status \"c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25\": rpc error: code = NotFound desc = could not find container \"c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25\": container with ID starting with c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25 not found: ID does not exist" Jul 17 17:52:08.987820 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.987764 2574 scope.go:117] "RemoveContainer" containerID="dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699" Jul 17 17:52:08.988108 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:52:08.988080 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699\": container with ID starting with dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699 not found: ID does not exist" containerID="dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699" Jul 17 17:52:08.988204 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.988121 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699"} err="failed to get container status \"dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699\": rpc error: code = NotFound desc = could not find container \"dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699\": container with ID starting with dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699 not found: ID does not exist" Jul 17 17:52:08.988204 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.988145 2574 scope.go:117] "RemoveContainer" containerID="6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031" Jul 17 17:52:08.988492 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:52:08.988465 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031\": container with ID starting with 6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031 not found: ID does not exist" containerID="6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031" Jul 17 17:52:08.988560 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.988504 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031"} err="failed to get container status \"6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031\": rpc error: code = NotFound desc = could not find container \"6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031\": container with ID starting with 6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031 not found: ID does not exist" Jul 17 17:52:08.988560 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.988529 2574 scope.go:117] "RemoveContainer" containerID="f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488" Jul 17 17:52:08.988664 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.988571 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b978bdb1-82ea-40a0-b843-a148a14e5c22" path="/var/lib/kubelet/pods/b978bdb1-82ea-40a0-b843-a148a14e5c22/volumes" Jul 17 17:52:08.988861 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.988824 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 17 17:52:08.989028 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:52:08.989006 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488\": container with ID starting with f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488 not found: ID does not exist" containerID="f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488" Jul 17 17:52:08.989122 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.989035 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488"} err="failed to get container status \"f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488\": rpc error: code = NotFound desc = could not find container \"f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488\": container with ID starting with f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488 not found: ID does not exist" Jul 17 17:52:08.989122 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.989055 2574 scope.go:117] "RemoveContainer" containerID="5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20" Jul 17 17:52:08.989415 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:52:08.989391 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20\": container with ID starting with 5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20 not found: ID does not exist" containerID="5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20" Jul 17 17:52:08.989513 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.989423 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20"} err="failed to get container status \"5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20\": rpc error: code = NotFound desc = could not find container \"5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20\": container with ID starting with 5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20 not found: ID does not exist" Jul 17 17:52:08.989513 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.989472 2574 scope.go:117] "RemoveContainer" containerID="4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7" Jul 17 17:52:08.989999 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:52:08.989936 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7\": container with ID starting with 4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7 not found: ID does not exist" containerID="4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7" Jul 17 17:52:08.990108 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.990006 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 17 17:52:08.990108 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.990004 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7"} err="failed to get container status \"4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7\": rpc error: code = NotFound desc = could not find container \"4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7\": container with ID starting with 4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7 not found: ID does not exist" Jul 17 17:52:08.990108 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.990028 2574 scope.go:117] "RemoveContainer" containerID="f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42" Jul 17 17:52:08.991034 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:52:08.991011 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42\": container with ID starting with f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42 not found: ID does not exist" containerID="f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42" Jul 17 17:52:08.991103 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.991037 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42"} err="failed to get container status \"f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42\": rpc error: code = NotFound desc = could not find container \"f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42\": container with ID starting with f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42 not found: ID does not exist" Jul 17 17:52:08.991103 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.991058 2574 scope.go:117] "RemoveContainer" containerID="c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25" Jul 17 17:52:08.991365 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.991332 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25"} err="failed to get container status \"c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25\": rpc error: code = NotFound desc = could not find container \"c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25\": container with ID starting with c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25 not found: ID does not exist" Jul 17 17:52:08.991436 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.991376 2574 scope.go:117] "RemoveContainer" containerID="dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699" Jul 17 17:52:08.991684 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.991660 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699"} err="failed to get container status \"dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699\": rpc error: code = NotFound desc = could not find container \"dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699\": container with ID starting with dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699 not found: ID does not exist" Jul 17 17:52:08.991684 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.991681 2574 scope.go:117] "RemoveContainer" containerID="6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031" Jul 17 17:52:08.992012 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.991990 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031"} err="failed to get container status \"6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031\": rpc error: code = NotFound desc = could not find container \"6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031\": container with ID starting with 6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031 not found: ID does not exist" Jul 17 17:52:08.992074 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.992014 2574 scope.go:117] "RemoveContainer" containerID="f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488" Jul 17 17:52:08.992264 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.992246 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488"} err="failed to get container status \"f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488\": rpc error: code = NotFound desc = could not find container \"f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488\": container with ID starting with f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488 not found: ID does not exist" Jul 17 17:52:08.992318 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.992265 2574 scope.go:117] "RemoveContainer" containerID="5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20" Jul 17 17:52:08.992555 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.992536 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20"} err="failed to get container status \"5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20\": rpc error: code = NotFound desc = could not find container \"5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20\": container with ID starting with 5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20 not found: ID does not exist" Jul 17 17:52:08.992626 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.992557 2574 scope.go:117] "RemoveContainer" containerID="4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7" Jul 17 17:52:08.992831 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.992814 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7"} err="failed to get container status \"4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7\": rpc error: code = NotFound desc = could not find container \"4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7\": container with ID starting with 4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7 not found: ID does not exist" Jul 17 17:52:08.992891 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.992833 2574 scope.go:117] "RemoveContainer" containerID="f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42" Jul 17 17:52:08.993074 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.993054 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42"} err="failed to get container status \"f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42\": rpc error: code = NotFound desc = could not find container \"f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42\": container with ID starting with f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42 not found: ID does not exist" Jul 17 17:52:08.993151 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.993075 2574 scope.go:117] "RemoveContainer" containerID="c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25" Jul 17 17:52:08.993386 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.993339 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25"} err="failed to get container status \"c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25\": rpc error: code = NotFound desc = could not find container \"c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25\": container with ID starting with c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25 not found: ID does not exist" Jul 17 17:52:08.993386 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.993384 2574 scope.go:117] "RemoveContainer" containerID="dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699" Jul 17 17:52:08.993525 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.993513 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:52:08.993661 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.993637 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699"} err="failed to get container status \"dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699\": rpc error: code = NotFound desc = could not find container \"dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699\": container with ID starting with dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699 not found: ID does not exist" Jul 17 17:52:08.993727 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.993663 2574 scope.go:117] "RemoveContainer" containerID="6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031" Jul 17 17:52:08.993894 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.993873 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031"} err="failed to get container status \"6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031\": rpc error: code = NotFound desc = could not find container \"6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031\": container with ID starting with 6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031 not found: ID does not exist" Jul 17 17:52:08.993957 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.993893 2574 scope.go:117] "RemoveContainer" containerID="f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488" Jul 17 17:52:08.994149 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.994109 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488"} err="failed to get container status \"f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488\": rpc error: code = NotFound desc = could not find container \"f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488\": container with ID starting with f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488 not found: ID does not exist" Jul 17 17:52:08.994149 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.994133 2574 scope.go:117] "RemoveContainer" containerID="5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20" Jul 17 17:52:08.994439 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.994416 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20"} err="failed to get container status \"5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20\": rpc error: code = NotFound desc = could not find container \"5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20\": container with ID starting with 5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20 not found: ID does not exist" Jul 17 17:52:08.994528 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.994441 2574 scope.go:117] "RemoveContainer" containerID="4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7" Jul 17 17:52:08.994806 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.994716 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7"} err="failed to get container status \"4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7\": rpc error: code = NotFound desc = could not find container \"4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7\": container with ID starting with 4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7 not found: ID does not exist" Jul 17 17:52:08.994806 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.994743 2574 scope.go:117] "RemoveContainer" containerID="f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42" Jul 17 17:52:08.994990 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.994970 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42"} err="failed to get container status \"f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42\": rpc error: code = NotFound desc = could not find container \"f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42\": container with ID starting with f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42 not found: ID does not exist" Jul 17 17:52:08.995032 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.994994 2574 scope.go:117] "RemoveContainer" containerID="c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25" Jul 17 17:52:08.995230 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.995210 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25"} err="failed to get container status \"c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25\": rpc error: code = NotFound desc = could not find container \"c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25\": container with ID starting with c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25 not found: ID does not exist" Jul 17 17:52:08.995310 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.995232 2574 scope.go:117] "RemoveContainer" containerID="dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699" Jul 17 17:52:08.995470 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.995441 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699"} err="failed to get container status \"dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699\": rpc error: code = NotFound desc = could not find container \"dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699\": container with ID starting with dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699 not found: ID does not exist" Jul 17 17:52:08.995537 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.995472 2574 scope.go:117] "RemoveContainer" containerID="6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031" Jul 17 17:52:08.995692 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.995675 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031"} err="failed to get container status \"6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031\": rpc error: code = NotFound desc = could not find container \"6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031\": container with ID starting with 6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031 not found: ID does not exist" Jul 17 17:52:08.995762 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.995693 2574 scope.go:117] "RemoveContainer" containerID="f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488" Jul 17 17:52:08.996158 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.996133 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488"} err="failed to get container status \"f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488\": rpc error: code = NotFound desc = could not find container \"f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488\": container with ID starting with f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488 not found: ID does not exist" Jul 17 17:52:08.996248 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.996159 2574 scope.go:117] "RemoveContainer" containerID="5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20" Jul 17 17:52:08.996482 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.996462 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20"} err="failed to get container status \"5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20\": rpc error: code = NotFound desc = could not find container \"5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20\": container with ID starting with 5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20 not found: ID does not exist" Jul 17 17:52:08.996571 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.996484 2574 scope.go:117] "RemoveContainer" containerID="4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7" Jul 17 17:52:08.996823 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.996803 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7"} err="failed to get container status \"4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7\": rpc error: code = NotFound desc = could not find container \"4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7\": container with ID starting with 4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7 not found: ID does not exist" Jul 17 17:52:08.996934 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.996923 2574 scope.go:117] "RemoveContainer" containerID="f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42" Jul 17 17:52:08.997252 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.997222 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42"} err="failed to get container status \"f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42\": rpc error: code = NotFound desc = could not find container \"f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42\": container with ID starting with f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42 not found: ID does not exist" Jul 17 17:52:08.997380 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.997343 2574 scope.go:117] "RemoveContainer" containerID="c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25" Jul 17 17:52:08.997709 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.997689 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25"} err="failed to get container status \"c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25\": rpc error: code = NotFound desc = could not find container \"c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25\": container with ID starting with c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25 not found: ID does not exist" Jul 17 17:52:08.997806 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.997796 2574 scope.go:117] "RemoveContainer" containerID="dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699" Jul 17 17:52:08.998135 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.998110 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699"} err="failed to get container status \"dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699\": rpc error: code = NotFound desc = could not find container \"dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699\": container with ID starting with dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699 not found: ID does not exist" Jul 17 17:52:08.998135 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.998134 2574 scope.go:117] "RemoveContainer" containerID="6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031" Jul 17 17:52:08.998415 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.998396 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031"} err="failed to get container status \"6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031\": rpc error: code = NotFound desc = could not find container \"6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031\": container with ID starting with 6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031 not found: ID does not exist" Jul 17 17:52:08.998493 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.998417 2574 scope.go:117] "RemoveContainer" containerID="f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488" Jul 17 17:52:08.998669 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.998649 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488"} err="failed to get container status \"f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488\": rpc error: code = NotFound desc = could not find container \"f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488\": container with ID starting with f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488 not found: ID does not exist" Jul 17 17:52:08.998669 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.998668 2574 scope.go:117] "RemoveContainer" containerID="5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20" Jul 17 17:52:08.998907 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.998891 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20"} err="failed to get container status \"5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20\": rpc error: code = NotFound desc = could not find container \"5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20\": container with ID starting with 5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20 not found: ID does not exist" Jul 17 17:52:08.998948 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.998907 2574 scope.go:117] "RemoveContainer" containerID="4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7" Jul 17 17:52:08.999111 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.999095 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7"} err="failed to get container status \"4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7\": rpc error: code = NotFound desc = could not find container \"4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7\": container with ID starting with 4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7 not found: ID does not exist" Jul 17 17:52:08.999111 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.999111 2574 scope.go:117] "RemoveContainer" containerID="f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42" Jul 17 17:52:08.999310 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.999295 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42"} err="failed to get container status \"f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42\": rpc error: code = NotFound desc = could not find container \"f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42\": container with ID starting with f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42 not found: ID does not exist" Jul 17 17:52:08.999387 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.999310 2574 scope.go:117] "RemoveContainer" containerID="c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25" Jul 17 17:52:08.999529 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.999512 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25"} err="failed to get container status \"c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25\": rpc error: code = NotFound desc = could not find container \"c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25\": container with ID starting with c7a07f8d5e14b09353ace850d6b243084b6c469aa9e89e98b1e25184eb4b4d25 not found: ID does not exist" Jul 17 17:52:08.999569 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.999529 2574 scope.go:117] "RemoveContainer" containerID="dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699" Jul 17 17:52:08.999741 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.999726 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699"} err="failed to get container status \"dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699\": rpc error: code = NotFound desc = could not find container \"dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699\": container with ID starting with dc5c2c5eaceff16b26b93ab304bcca369945060b4fd55a39f90e373a7556f699 not found: ID does not exist" Jul 17 17:52:08.999783 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.999741 2574 scope.go:117] "RemoveContainer" containerID="6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031" Jul 17 17:52:08.999940 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.999924 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031"} err="failed to get container status \"6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031\": rpc error: code = NotFound desc = could not find container \"6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031\": container with ID starting with 6c526ef663f8d65dd35904b7c4fb7c58fdabdf6800b4217269a576ec72656031 not found: ID does not exist" Jul 17 17:52:08.999982 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:08.999940 2574 scope.go:117] "RemoveContainer" containerID="f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488" Jul 17 17:52:09.000139 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.000125 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488"} err="failed to get container status \"f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488\": rpc error: code = NotFound desc = could not find container \"f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488\": container with ID starting with f8ece2767b6df7b279e134737cdfb656afb98d9bfc80d53ad67661f08d425488 not found: ID does not exist" Jul 17 17:52:09.000180 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.000139 2574 scope.go:117] "RemoveContainer" containerID="5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20" Jul 17 17:52:09.000322 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.000307 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20"} err="failed to get container status \"5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20\": rpc error: code = NotFound desc = could not find container \"5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20\": container with ID starting with 5a884b1f417324d63f4e8c0797c50f01fd098bd4e06ee51e88c10b8e8adaef20 not found: ID does not exist" Jul 17 17:52:09.000396 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.000322 2574 scope.go:117] "RemoveContainer" containerID="4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7" Jul 17 17:52:09.000524 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.000507 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7"} err="failed to get container status \"4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7\": rpc error: code = NotFound desc = could not find container \"4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7\": container with ID starting with 4f40e14c5781fceab2a1abc029c5cd079a9f234ea686768f707777e3acfbecf7 not found: ID does not exist" Jul 17 17:52:09.000570 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.000523 2574 scope.go:117] "RemoveContainer" containerID="f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42" Jul 17 17:52:09.000739 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.000719 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42"} err="failed to get container status \"f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42\": rpc error: code = NotFound desc = could not find container \"f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42\": container with ID starting with f0d1ebab98665be320da984d455120e066e7782a46c7a175b57b210b15ed3b42 not found: ID does not exist" Jul 17 17:52:09.068862 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.068823 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.068862 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.068864 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zhvj\" (UniqueName: \"kubernetes.io/projected/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-kube-api-access-9zhvj\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.069073 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.068886 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.069073 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.068966 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-config-out\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.069073 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.068997 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.069073 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.069024 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.069192 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.069072 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.069192 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.069108 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.069192 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.069156 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-config\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.069192 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.069174 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.069315 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.069190 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.069315 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.069211 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-web-config\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.069315 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.069285 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.069454 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.069313 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.069454 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.069339 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.069454 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.069378 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.069454 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.069411 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.069576 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.069470 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.069576 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.069493 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.069576 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.069509 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.170389 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.170256 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.170389 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.170310 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zhvj\" (UniqueName: \"kubernetes.io/projected/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-kube-api-access-9zhvj\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.170389 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.170333 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.170389 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.170378 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-config-out\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.170832 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.170489 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.170832 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.170527 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.170832 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.170555 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.170832 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.170596 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.170832 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.170627 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-config\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.170832 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.170651 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.170832 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.170693 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.170832 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.170722 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-web-config\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.170832 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.170769 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.170832 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.170806 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.170832 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.170835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.171593 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.170858 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.171593 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.170888 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.171593 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.170945 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.171593 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.170984 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.171593 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.171008 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.171593 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.171121 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.171593 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.171285 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.172136 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.172111 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.172444 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.172422 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.173038 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.173012 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.173452 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.173430 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-config-out\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.173553 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.173530 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.173776 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.173614 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.173776 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.173670 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.173776 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.173731 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.174129 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.174103 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.174202 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.174104 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.174646 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.174621 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.175522 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.175501 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.175936 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.175918 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-config\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.176191 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.176168 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-web-config\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.176493 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.176474 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.176493 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.176486 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.177230 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.177211 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.178468 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.178448 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zhvj\" (UniqueName: \"kubernetes.io/projected/04402fe8-4d27-4800-b9c4-ebccfa4a2c4f-kube-api-access-9zhvj\") pod \"prometheus-k8s-0\" (UID: \"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.294465 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.294428 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:09.437742 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.437533 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 17 17:52:09.440090 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:52:09.440048 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04402fe8_4d27_4800_b9c4_ebccfa4a2c4f.slice/crio-030ce1b0d1461ccc7f138ae3f81f82b7379bfa24d77216490a04340e705a79de WatchSource:0}: Error finding container 030ce1b0d1461ccc7f138ae3f81f82b7379bfa24d77216490a04340e705a79de: Status 404 returned error can't find the container with id 030ce1b0d1461ccc7f138ae3f81f82b7379bfa24d77216490a04340e705a79de Jul 17 17:52:09.927235 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.927201 2574 generic.go:356] "Generic (PLEG): container finished" podID="04402fe8-4d27-4800-b9c4-ebccfa4a2c4f" containerID="9c46e96020c9ef5a4a2312f83e496f9acc42ec96ff5da6b0fe375b2c68056f76" exitCode=0 Jul 17 17:52:09.927720 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.927290 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f","Type":"ContainerDied","Data":"9c46e96020c9ef5a4a2312f83e496f9acc42ec96ff5da6b0fe375b2c68056f76"} Jul 17 17:52:09.927720 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:09.927402 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f","Type":"ContainerStarted","Data":"030ce1b0d1461ccc7f138ae3f81f82b7379bfa24d77216490a04340e705a79de"} Jul 17 17:52:10.935049 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:10.935017 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f","Type":"ContainerStarted","Data":"f4bb83c9b170f049d1814df77acfcc5ed9d6db89699a880a374e68da9e321a20"} Jul 17 17:52:10.935534 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:10.935106 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f","Type":"ContainerStarted","Data":"5bfbe734a79b780633c80d3eb959c65264fc39523d6b36ef96e530fc52008b77"} Jul 17 17:52:10.935534 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:10.935134 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f","Type":"ContainerStarted","Data":"2ed26cd98146f64408b1030d470b534a3748149abc4222b1dc23e4e553478161"} Jul 17 17:52:10.935534 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:10.935158 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f","Type":"ContainerStarted","Data":"fe812a5e05423be318b9ac9c9b11bbb0d7873a278d675cd54fe4312c71ffe6df"} Jul 17 17:52:10.935534 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:10.935184 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f","Type":"ContainerStarted","Data":"5aa97024ffc514b85fa8dc3ceb99775be03c85350c9a8763f07f3515b974f4a4"} Jul 17 17:52:10.935534 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:10.935206 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04402fe8-4d27-4800-b9c4-ebccfa4a2c4f","Type":"ContainerStarted","Data":"f7ce91dbb01ac3593a9b2aca6e4c5096057ee9a662e62dfcefe4b8ec05a0fee7"} Jul 17 17:52:10.965507 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:10.965440 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.939631219 podStartE2EDuration="2.965411745s" totalImagesPullingTime="0s" totalInitContainerRuntime="25.780526ms" isStatefulPod=false podCreationTimestamp="2026-07-17 17:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:52:10.963616709 +0000 UTC m=+142.551569398" watchObservedRunningTime="2026-07-17 17:52:10.965411745 +0000 UTC m=+142.553364436" Jul 17 17:52:11.525523 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:11.525487 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:52:11.625835 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:11.625798 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:52:14.294585 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:14.294549 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:52:17.547213 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:17.547173 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:52:17.603140 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:17.603106 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:52:17.659736 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:17.659708 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:52:20.562428 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:20.562389 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:52:20.663603 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:20.663575 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:52:20.763801 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:20.763769 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:52:44.797106 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:44.797067 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:52:44.846463 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:52:44.846432 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:53:03.692565 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:03.692531 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:53:03.793454 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:03.793425 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:53:09.295637 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:09.295591 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:53:09.312594 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:09.312565 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:53:10.196471 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:10.196442 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 17 17:53:10.791557 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:10.791522 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:53:12.969544 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:12.969513 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:53:25.315518 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:25.315486 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:53:26.099214 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:26.099178 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:53:28.912805 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:28.912774 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:53:28.970257 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:28.970223 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:53:30.896530 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:30.896496 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:53:39.073766 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:39.073725 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sgtmk"] Jul 17 17:53:39.077276 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:39.077254 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-sgtmk" Jul 17 17:53:39.079612 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:39.079589 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 17 17:53:39.079855 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:39.079842 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 17 17:53:39.080833 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:39.080820 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-b2dr4\"" Jul 17 17:53:39.084661 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:39.084459 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sgtmk"] Jul 17 17:53:39.184556 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:39.184510 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qtj9\" (UniqueName: \"kubernetes.io/projected/2baf1c4e-31ec-4beb-8718-937527b43642-kube-api-access-7qtj9\") pod \"cert-manager-webhook-6486ffd997-sgtmk\" (UID: \"2baf1c4e-31ec-4beb-8718-937527b43642\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sgtmk" Jul 17 17:53:39.184733 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:39.184575 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2baf1c4e-31ec-4beb-8718-937527b43642-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sgtmk\" (UID: \"2baf1c4e-31ec-4beb-8718-937527b43642\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sgtmk" Jul 17 17:53:39.285970 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:39.285933 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qtj9\" (UniqueName: \"kubernetes.io/projected/2baf1c4e-31ec-4beb-8718-937527b43642-kube-api-access-7qtj9\") pod \"cert-manager-webhook-6486ffd997-sgtmk\" (UID: \"2baf1c4e-31ec-4beb-8718-937527b43642\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sgtmk" Jul 17 17:53:39.286167 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:39.285984 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2baf1c4e-31ec-4beb-8718-937527b43642-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sgtmk\" (UID: \"2baf1c4e-31ec-4beb-8718-937527b43642\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sgtmk" Jul 17 17:53:39.293846 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:39.293822 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2baf1c4e-31ec-4beb-8718-937527b43642-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sgtmk\" (UID: \"2baf1c4e-31ec-4beb-8718-937527b43642\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sgtmk" Jul 17 17:53:39.294112 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:39.294090 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qtj9\" (UniqueName: \"kubernetes.io/projected/2baf1c4e-31ec-4beb-8718-937527b43642-kube-api-access-7qtj9\") pod \"cert-manager-webhook-6486ffd997-sgtmk\" (UID: \"2baf1c4e-31ec-4beb-8718-937527b43642\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sgtmk" Jul 17 17:53:39.396674 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:39.396587 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-sgtmk" Jul 17 17:53:39.533238 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:39.533216 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sgtmk"] Jul 17 17:53:39.535933 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:53:39.535902 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2baf1c4e_31ec_4beb_8718_937527b43642.slice/crio-71b7317f27a34b35286aa0b488d4ed332de16f16e7b25501f183a851a3a79e2f WatchSource:0}: Error finding container 71b7317f27a34b35286aa0b488d4ed332de16f16e7b25501f183a851a3a79e2f: Status 404 returned error can't find the container with id 71b7317f27a34b35286aa0b488d4ed332de16f16e7b25501f183a851a3a79e2f Jul 17 17:53:40.312632 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:40.312586 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-sgtmk" event={"ID":"2baf1c4e-31ec-4beb-8718-937527b43642","Type":"ContainerStarted","Data":"71b7317f27a34b35286aa0b488d4ed332de16f16e7b25501f183a851a3a79e2f"} Jul 17 17:53:40.957982 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:40.957923 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:53:41.010795 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:41.010759 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:53:41.301286 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:41.301249 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:53:43.329842 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:43.329805 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-sgtmk" event={"ID":"2baf1c4e-31ec-4beb-8718-937527b43642","Type":"ContainerStarted","Data":"e85c2cb7d260baa0258880df8812e12ac9a9e09bd78ae527b27ed4d9a9705515"} Jul 17 17:53:43.330249 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:43.329950 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-sgtmk" Jul 17 17:53:43.346719 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:43.346672 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-sgtmk" podStartSLOduration=1.237163176 podStartE2EDuration="4.346655963s" totalImagesPullingTime="3.109492787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:53:43.345288292 +0000 UTC m=+234.933240983" watchObservedRunningTime="2026-07-17 17:53:43.346655963 +0000 UTC m=+234.934608652" Jul 17 17:53:43.950203 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:43.950168 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:53:49.337187 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:49.337156 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-sgtmk" Jul 17 17:53:53.437741 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:53.437708 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:53:53.537609 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:53.537573 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:53:55.107971 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:55.107936 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fxhg4"] Jul 17 17:53:55.111552 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:55.111535 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fxhg4" Jul 17 17:53:55.114431 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:55.114408 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-7j8mx\"" Jul 17 17:53:55.115302 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:55.115282 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 17 17:53:55.115440 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:55.115325 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 17 17:53:55.119794 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:55.119772 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fxhg4"] Jul 17 17:53:55.236909 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:55.236873 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb8d98a3-eae2-4640-8a32-b56c4a9b702a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fxhg4\" (UID: \"bb8d98a3-eae2-4640-8a32-b56c4a9b702a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fxhg4" Jul 17 17:53:55.237111 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:55.236918 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r8mm\" (UniqueName: \"kubernetes.io/projected/bb8d98a3-eae2-4640-8a32-b56c4a9b702a-kube-api-access-9r8mm\") pod \"openshift-lws-operator-fd8ccff4c-fxhg4\" (UID: \"bb8d98a3-eae2-4640-8a32-b56c4a9b702a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fxhg4" Jul 17 17:53:55.338085 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:55.338043 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb8d98a3-eae2-4640-8a32-b56c4a9b702a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fxhg4\" (UID: \"bb8d98a3-eae2-4640-8a32-b56c4a9b702a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fxhg4" Jul 17 17:53:55.338085 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:55.338084 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r8mm\" (UniqueName: \"kubernetes.io/projected/bb8d98a3-eae2-4640-8a32-b56c4a9b702a-kube-api-access-9r8mm\") pod \"openshift-lws-operator-fd8ccff4c-fxhg4\" (UID: \"bb8d98a3-eae2-4640-8a32-b56c4a9b702a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fxhg4" Jul 17 17:53:55.338530 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:55.338513 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb8d98a3-eae2-4640-8a32-b56c4a9b702a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fxhg4\" (UID: \"bb8d98a3-eae2-4640-8a32-b56c4a9b702a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fxhg4" Jul 17 17:53:55.346555 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:55.346532 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r8mm\" (UniqueName: \"kubernetes.io/projected/bb8d98a3-eae2-4640-8a32-b56c4a9b702a-kube-api-access-9r8mm\") pod \"openshift-lws-operator-fd8ccff4c-fxhg4\" (UID: \"bb8d98a3-eae2-4640-8a32-b56c4a9b702a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fxhg4" Jul 17 17:53:55.421690 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:55.421593 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fxhg4" Jul 17 17:53:55.556327 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:55.556299 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fxhg4"] Jul 17 17:53:55.559202 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:53:55.559180 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb8d98a3_eae2_4640_8a32_b56c4a9b702a.slice/crio-4220593ea041a5c365208e812a253819ca0f89b505bd8ade6fd02dd518a11d5d WatchSource:0}: Error finding container 4220593ea041a5c365208e812a253819ca0f89b505bd8ade6fd02dd518a11d5d: Status 404 returned error can't find the container with id 4220593ea041a5c365208e812a253819ca0f89b505bd8ade6fd02dd518a11d5d Jul 17 17:53:56.389098 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:56.389057 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fxhg4" event={"ID":"bb8d98a3-eae2-4640-8a32-b56c4a9b702a","Type":"ContainerStarted","Data":"4220593ea041a5c365208e812a253819ca0f89b505bd8ade6fd02dd518a11d5d"} Jul 17 17:53:58.399560 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:58.399468 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fxhg4" event={"ID":"bb8d98a3-eae2-4640-8a32-b56c4a9b702a","Type":"ContainerStarted","Data":"ef886a5272578fb611ec23c2c21b19c18c6d42a44d6fcccad2c16d4580931aa8"} Jul 17 17:53:58.416451 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:58.416391 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fxhg4" podStartSLOduration=0.926373936 podStartE2EDuration="3.41634876s" totalImagesPullingTime="2.489974824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:53:58.415761611 +0000 UTC m=+250.003714301" watchObservedRunningTime="2026-07-17 17:53:58.41634876 +0000 UTC m=+250.004301451" Jul 17 17:53:58.758301 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:58.758267 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:53:58.853349 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:58.853306 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:53:59.073623 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:59.073535 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:53:59.173638 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:53:59.173601 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:01.211903 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:01.211874 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:05.439331 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.439243 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:05.491022 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.490988 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6568db77d8-ldt7z"] Jul 17 17:54:05.494725 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.494703 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6568db77d8-ldt7z" Jul 17 17:54:05.497509 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.497487 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 17 17:54:05.498686 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.498668 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 17 17:54:05.498771 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.498706 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 17 17:54:05.498834 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.498709 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jzlsr\"" Jul 17 17:54:05.506038 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.506016 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6568db77d8-ldt7z"] Jul 17 17:54:05.527666 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.527626 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:05.527666 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.527631 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmlkl\" (UniqueName: \"kubernetes.io/projected/a7c57b39-9500-4bce-bda2-b5ab525d97d6-kube-api-access-hmlkl\") pod \"lws-controller-manager-6568db77d8-ldt7z\" (UID: \"a7c57b39-9500-4bce-bda2-b5ab525d97d6\") " pod="openshift-lws-operator/lws-controller-manager-6568db77d8-ldt7z" Jul 17 17:54:05.527666 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.527669 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a7c57b39-9500-4bce-bda2-b5ab525d97d6-manager-config\") pod \"lws-controller-manager-6568db77d8-ldt7z\" (UID: \"a7c57b39-9500-4bce-bda2-b5ab525d97d6\") " pod="openshift-lws-operator/lws-controller-manager-6568db77d8-ldt7z" Jul 17 17:54:05.527941 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.527708 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7c57b39-9500-4bce-bda2-b5ab525d97d6-cert\") pod \"lws-controller-manager-6568db77d8-ldt7z\" (UID: \"a7c57b39-9500-4bce-bda2-b5ab525d97d6\") " pod="openshift-lws-operator/lws-controller-manager-6568db77d8-ldt7z" Jul 17 17:54:05.527941 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.527813 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7c57b39-9500-4bce-bda2-b5ab525d97d6-metrics-cert\") pod \"lws-controller-manager-6568db77d8-ldt7z\" (UID: \"a7c57b39-9500-4bce-bda2-b5ab525d97d6\") " pod="openshift-lws-operator/lws-controller-manager-6568db77d8-ldt7z" Jul 17 17:54:05.629107 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.629066 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7c57b39-9500-4bce-bda2-b5ab525d97d6-metrics-cert\") pod \"lws-controller-manager-6568db77d8-ldt7z\" (UID: \"a7c57b39-9500-4bce-bda2-b5ab525d97d6\") " pod="openshift-lws-operator/lws-controller-manager-6568db77d8-ldt7z" Jul 17 17:54:05.629107 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.629112 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmlkl\" (UniqueName: \"kubernetes.io/projected/a7c57b39-9500-4bce-bda2-b5ab525d97d6-kube-api-access-hmlkl\") pod \"lws-controller-manager-6568db77d8-ldt7z\" (UID: \"a7c57b39-9500-4bce-bda2-b5ab525d97d6\") " pod="openshift-lws-operator/lws-controller-manager-6568db77d8-ldt7z" Jul 17 17:54:05.629412 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.629133 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a7c57b39-9500-4bce-bda2-b5ab525d97d6-manager-config\") pod \"lws-controller-manager-6568db77d8-ldt7z\" (UID: \"a7c57b39-9500-4bce-bda2-b5ab525d97d6\") " pod="openshift-lws-operator/lws-controller-manager-6568db77d8-ldt7z" Jul 17 17:54:05.629412 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.629160 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7c57b39-9500-4bce-bda2-b5ab525d97d6-cert\") pod \"lws-controller-manager-6568db77d8-ldt7z\" (UID: \"a7c57b39-9500-4bce-bda2-b5ab525d97d6\") " pod="openshift-lws-operator/lws-controller-manager-6568db77d8-ldt7z" Jul 17 17:54:05.629867 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.629842 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a7c57b39-9500-4bce-bda2-b5ab525d97d6-manager-config\") pod \"lws-controller-manager-6568db77d8-ldt7z\" (UID: \"a7c57b39-9500-4bce-bda2-b5ab525d97d6\") " pod="openshift-lws-operator/lws-controller-manager-6568db77d8-ldt7z" Jul 17 17:54:05.631776 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.631746 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7c57b39-9500-4bce-bda2-b5ab525d97d6-metrics-cert\") pod \"lws-controller-manager-6568db77d8-ldt7z\" (UID: \"a7c57b39-9500-4bce-bda2-b5ab525d97d6\") " pod="openshift-lws-operator/lws-controller-manager-6568db77d8-ldt7z" Jul 17 17:54:05.631858 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.631777 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7c57b39-9500-4bce-bda2-b5ab525d97d6-cert\") pod \"lws-controller-manager-6568db77d8-ldt7z\" (UID: \"a7c57b39-9500-4bce-bda2-b5ab525d97d6\") " pod="openshift-lws-operator/lws-controller-manager-6568db77d8-ldt7z" Jul 17 17:54:05.643862 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.643838 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmlkl\" (UniqueName: \"kubernetes.io/projected/a7c57b39-9500-4bce-bda2-b5ab525d97d6-kube-api-access-hmlkl\") pod \"lws-controller-manager-6568db77d8-ldt7z\" (UID: \"a7c57b39-9500-4bce-bda2-b5ab525d97d6\") " pod="openshift-lws-operator/lws-controller-manager-6568db77d8-ldt7z" Jul 17 17:54:05.805296 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.805253 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6568db77d8-ldt7z" Jul 17 17:54:05.941133 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:05.941106 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6568db77d8-ldt7z"] Jul 17 17:54:05.943788 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:54:05.943761 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c57b39_9500_4bce_bda2_b5ab525d97d6.slice/crio-b6a00ff39407ec0e6081797b6e4a957ae79c95f3c734d12027c788a153d0bf91 WatchSource:0}: Error finding container b6a00ff39407ec0e6081797b6e4a957ae79c95f3c734d12027c788a153d0bf91: Status 404 returned error can't find the container with id b6a00ff39407ec0e6081797b6e4a957ae79c95f3c734d12027c788a153d0bf91 Jul 17 17:54:06.433496 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:06.433455 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6568db77d8-ldt7z" event={"ID":"a7c57b39-9500-4bce-bda2-b5ab525d97d6","Type":"ContainerStarted","Data":"b6a00ff39407ec0e6081797b6e4a957ae79c95f3c734d12027c788a153d0bf91"} Jul 17 17:54:08.443812 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:08.443701 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6568db77d8-ldt7z" event={"ID":"a7c57b39-9500-4bce-bda2-b5ab525d97d6","Type":"ContainerStarted","Data":"d59aed4a649e7774ca4f97eba29f78e04c31b008470f7dfe2d95140460ec74ea"} Jul 17 17:54:08.443812 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:08.443806 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6568db77d8-ldt7z" Jul 17 17:54:08.462556 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:08.462510 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6568db77d8-ldt7z" podStartSLOduration=1.407173277 podStartE2EDuration="3.462493576s" totalImagesPullingTime="2.055320299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:54:08.461321526 +0000 UTC m=+260.049274215" watchObservedRunningTime="2026-07-17 17:54:08.462493576 +0000 UTC m=+260.050446265" Jul 17 17:54:14.986061 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:14.985967 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:16.032651 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.032621 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp"] Jul 17 17:54:16.088329 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.088292 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp"] Jul 17 17:54:16.088525 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.088433 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" Jul 17 17:54:16.091971 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.091947 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 17 17:54:16.093125 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.093106 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 17 17:54:16.093253 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.093203 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 17 17:54:16.093253 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.093239 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xc9bm\"" Jul 17 17:54:16.093399 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.093288 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 17 17:54:16.121374 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.121308 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f8367df-920a-417e-8033-8bad870315c6-webhook-cert\") pod \"opendatahub-operator-controller-manager-cc99fdf4f-bmtvp\" (UID: \"7f8367df-920a-417e-8033-8bad870315c6\") " pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" Jul 17 17:54:16.121542 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.121454 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7f8367df-920a-417e-8033-8bad870315c6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cc99fdf4f-bmtvp\" (UID: \"7f8367df-920a-417e-8033-8bad870315c6\") " pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" Jul 17 17:54:16.121542 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.121484 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f8367df-920a-417e-8033-8bad870315c6-tmp\") pod \"opendatahub-operator-controller-manager-cc99fdf4f-bmtvp\" (UID: \"7f8367df-920a-417e-8033-8bad870315c6\") " pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" Jul 17 17:54:16.121542 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.121512 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmwvt\" (UniqueName: \"kubernetes.io/projected/7f8367df-920a-417e-8033-8bad870315c6-kube-api-access-xmwvt\") pod \"opendatahub-operator-controller-manager-cc99fdf4f-bmtvp\" (UID: \"7f8367df-920a-417e-8033-8bad870315c6\") " pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" Jul 17 17:54:16.121646 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.121574 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7f8367df-920a-417e-8033-8bad870315c6-manifests\") pod \"opendatahub-operator-controller-manager-cc99fdf4f-bmtvp\" (UID: \"7f8367df-920a-417e-8033-8bad870315c6\") " pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" Jul 17 17:54:16.164071 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.164031 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:16.222399 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.222335 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7f8367df-920a-417e-8033-8bad870315c6-manifests\") pod \"opendatahub-operator-controller-manager-cc99fdf4f-bmtvp\" (UID: \"7f8367df-920a-417e-8033-8bad870315c6\") " pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" Jul 17 17:54:16.222578 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.222413 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f8367df-920a-417e-8033-8bad870315c6-webhook-cert\") pod \"opendatahub-operator-controller-manager-cc99fdf4f-bmtvp\" (UID: \"7f8367df-920a-417e-8033-8bad870315c6\") " pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" Jul 17 17:54:16.222578 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.222460 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7f8367df-920a-417e-8033-8bad870315c6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cc99fdf4f-bmtvp\" (UID: \"7f8367df-920a-417e-8033-8bad870315c6\") " pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" Jul 17 17:54:16.222578 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.222477 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f8367df-920a-417e-8033-8bad870315c6-tmp\") pod \"opendatahub-operator-controller-manager-cc99fdf4f-bmtvp\" (UID: \"7f8367df-920a-417e-8033-8bad870315c6\") " pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" Jul 17 17:54:16.222578 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.222511 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmwvt\" (UniqueName: \"kubernetes.io/projected/7f8367df-920a-417e-8033-8bad870315c6-kube-api-access-xmwvt\") pod \"opendatahub-operator-controller-manager-cc99fdf4f-bmtvp\" (UID: \"7f8367df-920a-417e-8033-8bad870315c6\") " pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" Jul 17 17:54:16.222749 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.222727 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7f8367df-920a-417e-8033-8bad870315c6-manifests\") pod \"opendatahub-operator-controller-manager-cc99fdf4f-bmtvp\" (UID: \"7f8367df-920a-417e-8033-8bad870315c6\") " pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" Jul 17 17:54:16.224942 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.224918 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f8367df-920a-417e-8033-8bad870315c6-tmp\") pod \"opendatahub-operator-controller-manager-cc99fdf4f-bmtvp\" (UID: \"7f8367df-920a-417e-8033-8bad870315c6\") " pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" Jul 17 17:54:16.225194 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.225173 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7f8367df-920a-417e-8033-8bad870315c6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cc99fdf4f-bmtvp\" (UID: \"7f8367df-920a-417e-8033-8bad870315c6\") " pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" Jul 17 17:54:16.225244 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.225231 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f8367df-920a-417e-8033-8bad870315c6-webhook-cert\") pod \"opendatahub-operator-controller-manager-cc99fdf4f-bmtvp\" (UID: \"7f8367df-920a-417e-8033-8bad870315c6\") " pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" Jul 17 17:54:16.230747 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.230720 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmwvt\" (UniqueName: \"kubernetes.io/projected/7f8367df-920a-417e-8033-8bad870315c6-kube-api-access-xmwvt\") pod \"opendatahub-operator-controller-manager-cc99fdf4f-bmtvp\" (UID: \"7f8367df-920a-417e-8033-8bad870315c6\") " pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" Jul 17 17:54:16.260518 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.260485 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:16.360877 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.360798 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:16.398826 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.398794 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" Jul 17 17:54:16.462216 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.462187 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:16.563409 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.563381 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp"] Jul 17 17:54:16.565024 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.564990 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:16.565996 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:54:16.565967 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8367df_920a_417e_8033_8bad870315c6.slice/crio-99aabda2c3d8eaf73065daba793a7b628ae8c0da038b159f67555346af5f23c3 WatchSource:0}: Error finding container 99aabda2c3d8eaf73065daba793a7b628ae8c0da038b159f67555346af5f23c3: Status 404 returned error can't find the container with id 99aabda2c3d8eaf73065daba793a7b628ae8c0da038b159f67555346af5f23c3 Jul 17 17:54:16.664628 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:16.664538 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:17.486443 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:17.486398 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" event={"ID":"7f8367df-920a-417e-8033-8bad870315c6","Type":"ContainerStarted","Data":"99aabda2c3d8eaf73065daba793a7b628ae8c0da038b159f67555346af5f23c3"} Jul 17 17:54:19.450310 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:19.450285 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6568db77d8-ldt7z" Jul 17 17:54:20.507045 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:20.507009 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" event={"ID":"7f8367df-920a-417e-8033-8bad870315c6","Type":"ContainerStarted","Data":"503907674fabc42e0c45b337b1fb14accae9c55c99e1b01cb0ee35bf5856de82"} Jul 17 17:54:21.513541 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:21.513503 2574 generic.go:356] "Generic (PLEG): container finished" podID="7f8367df-920a-417e-8033-8bad870315c6" containerID="503907674fabc42e0c45b337b1fb14accae9c55c99e1b01cb0ee35bf5856de82" exitCode=0 Jul 17 17:54:21.513970 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:21.513595 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" event={"ID":"7f8367df-920a-417e-8033-8bad870315c6","Type":"ContainerDied","Data":"503907674fabc42e0c45b337b1fb14accae9c55c99e1b01cb0ee35bf5856de82"} Jul 17 17:54:22.519974 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:22.519938 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" event={"ID":"7f8367df-920a-417e-8033-8bad870315c6","Type":"ContainerStarted","Data":"e931187e3904363ba266d9da2bf783d03739993acdfab1e74797c9ad4fbcaac4"} Jul 17 17:54:22.520479 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:22.520100 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" Jul 17 17:54:22.542420 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:22.542343 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" podStartSLOduration=2.035153483 podStartE2EDuration="6.542325851s" totalImagesPullingTime="3.34034716s" totalInitContainerRuntime="1.166825208s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:54:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:54:22.540088104 +0000 UTC m=+274.128040812" watchObservedRunningTime="2026-07-17 17:54:22.542325851 +0000 UTC m=+274.130278541" Jul 17 17:54:27.764904 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:27.764866 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:27.864564 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:27.864530 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:33.526944 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:33.526913 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-cc99fdf4f-bmtvp" Jul 17 17:54:35.122216 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.122184 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:35.223758 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.223725 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:35.437530 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.437445 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:35.590288 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.590256 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6799b9db4b-nhjjk"] Jul 17 17:54:35.599900 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.599876 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6799b9db4b-nhjjk" Jul 17 17:54:35.602851 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.602817 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2hlm6\"" Jul 17 17:54:35.604972 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.604947 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 17 17:54:35.605106 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.605003 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 17 17:54:35.605106 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.605035 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 17 17:54:35.605276 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.605252 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 17 17:54:35.608099 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.607932 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6799b9db4b-nhjjk"] Jul 17 17:54:35.703949 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.703849 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb3196bd-adc3-43b7-a538-9575d48afa1a-tmp\") pod \"kube-auth-proxy-6799b9db4b-nhjjk\" (UID: \"eb3196bd-adc3-43b7-a538-9575d48afa1a\") " pod="openshift-ingress/kube-auth-proxy-6799b9db4b-nhjjk" Jul 17 17:54:35.703949 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.703942 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3196bd-adc3-43b7-a538-9575d48afa1a-tls-certs\") pod \"kube-auth-proxy-6799b9db4b-nhjjk\" (UID: \"eb3196bd-adc3-43b7-a538-9575d48afa1a\") " pod="openshift-ingress/kube-auth-proxy-6799b9db4b-nhjjk" Jul 17 17:54:35.704142 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.703972 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2c8h\" (UniqueName: \"kubernetes.io/projected/eb3196bd-adc3-43b7-a538-9575d48afa1a-kube-api-access-z2c8h\") pod \"kube-auth-proxy-6799b9db4b-nhjjk\" (UID: \"eb3196bd-adc3-43b7-a538-9575d48afa1a\") " pod="openshift-ingress/kube-auth-proxy-6799b9db4b-nhjjk" Jul 17 17:54:35.774231 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.774200 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:35.805060 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.805021 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3196bd-adc3-43b7-a538-9575d48afa1a-tls-certs\") pod \"kube-auth-proxy-6799b9db4b-nhjjk\" (UID: \"eb3196bd-adc3-43b7-a538-9575d48afa1a\") " pod="openshift-ingress/kube-auth-proxy-6799b9db4b-nhjjk" Jul 17 17:54:35.805231 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.805086 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2c8h\" (UniqueName: \"kubernetes.io/projected/eb3196bd-adc3-43b7-a538-9575d48afa1a-kube-api-access-z2c8h\") pod \"kube-auth-proxy-6799b9db4b-nhjjk\" (UID: \"eb3196bd-adc3-43b7-a538-9575d48afa1a\") " pod="openshift-ingress/kube-auth-proxy-6799b9db4b-nhjjk" Jul 17 17:54:35.805231 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.805194 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb3196bd-adc3-43b7-a538-9575d48afa1a-tmp\") pod \"kube-auth-proxy-6799b9db4b-nhjjk\" (UID: \"eb3196bd-adc3-43b7-a538-9575d48afa1a\") " pod="openshift-ingress/kube-auth-proxy-6799b9db4b-nhjjk" Jul 17 17:54:35.807636 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.807607 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb3196bd-adc3-43b7-a538-9575d48afa1a-tmp\") pod \"kube-auth-proxy-6799b9db4b-nhjjk\" (UID: \"eb3196bd-adc3-43b7-a538-9575d48afa1a\") " pod="openshift-ingress/kube-auth-proxy-6799b9db4b-nhjjk" Jul 17 17:54:35.807955 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.807934 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3196bd-adc3-43b7-a538-9575d48afa1a-tls-certs\") pod \"kube-auth-proxy-6799b9db4b-nhjjk\" (UID: \"eb3196bd-adc3-43b7-a538-9575d48afa1a\") " pod="openshift-ingress/kube-auth-proxy-6799b9db4b-nhjjk" Jul 17 17:54:35.818692 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.818651 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2c8h\" (UniqueName: \"kubernetes.io/projected/eb3196bd-adc3-43b7-a538-9575d48afa1a-kube-api-access-z2c8h\") pod \"kube-auth-proxy-6799b9db4b-nhjjk\" (UID: \"eb3196bd-adc3-43b7-a538-9575d48afa1a\") " pod="openshift-ingress/kube-auth-proxy-6799b9db4b-nhjjk" Jul 17 17:54:35.874368 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.874331 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:35.911525 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:35.911497 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6799b9db4b-nhjjk" Jul 17 17:54:36.054044 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:36.054015 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6799b9db4b-nhjjk"] Jul 17 17:54:36.056419 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:54:36.056392 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3196bd_adc3_43b7_a538_9575d48afa1a.slice/crio-8ac9e48cc868da0a28e6e7d77febe13373700243387946a5199016fdf94768d7 WatchSource:0}: Error finding container 8ac9e48cc868da0a28e6e7d77febe13373700243387946a5199016fdf94768d7: Status 404 returned error can't find the container with id 8ac9e48cc868da0a28e6e7d77febe13373700243387946a5199016fdf94768d7 Jul 17 17:54:36.241929 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:36.241885 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:36.401851 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:36.401811 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:36.498461 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:36.498282 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:36.583924 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:36.583889 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6799b9db4b-nhjjk" event={"ID":"eb3196bd-adc3-43b7-a538-9575d48afa1a","Type":"ContainerStarted","Data":"8ac9e48cc868da0a28e6e7d77febe13373700243387946a5199016fdf94768d7"} Jul 17 17:54:36.874097 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:36.873933 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:36.943709 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:36.943664 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:37.766391 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:37.766340 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:39.600497 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:39.600453 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6799b9db4b-nhjjk" event={"ID":"eb3196bd-adc3-43b7-a538-9575d48afa1a","Type":"ContainerStarted","Data":"2ee70cac89631737f7aebf2eb80fcddfdd2d6687f9e0cd170c235236fbca9531"} Jul 17 17:54:39.616046 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:39.615995 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6799b9db4b-nhjjk" podStartSLOduration=1.8968826079999999 podStartE2EDuration="4.61597922s" totalImagesPullingTime="2.719096612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:54:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:54:39.615255756 +0000 UTC m=+291.203208447" watchObservedRunningTime="2026-07-17 17:54:39.61597922 +0000 UTC m=+291.203931909" Jul 17 17:54:42.469537 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:42.469495 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:42.569081 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:42.569040 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:42.931591 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:42.931512 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:43.030676 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:43.030628 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:43.134823 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:43.134792 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:43.274391 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:43.274333 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:43.448765 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:43.448729 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:44.675293 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:44.675265 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:45.618654 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:45.618627 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:45.787116 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:45.787078 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:45.869154 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:45.869064 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:45.978443 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:45.978325 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:46.126141 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:46.126052 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:46.244128 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:46.244092 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:46.353693 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:46.353666 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:54:48.862289 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:54:48.862259 2574 kubelet.go:1648] "Image garbage collection succeeded" Jul 17 17:55:36.610948 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:55:36.610914 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:55:36.707232 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:55:36.707197 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:55:36.806180 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:55:36.806143 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:55:47.281294 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:55:47.281255 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:05.705568 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:05.705536 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:09.906605 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:09.906570 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:10.281062 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.281025 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cc68cc5c9-gs95r"] Jul 17 17:56:10.284562 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.284540 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.287101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.287075 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 17 17:56:10.287101 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.287073 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 17 17:56:10.288236 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.288217 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 17 17:56:10.288394 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.288242 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ml5rl\"" Jul 17 17:56:10.288394 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.288330 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 17 17:56:10.288394 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.288348 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 17 17:56:10.292933 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.292909 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/affdc3b9-a61c-4b68-9436-7f3846d48355-service-ca\") pod \"console-7cc68cc5c9-gs95r\" (UID: \"affdc3b9-a61c-4b68-9436-7f3846d48355\") " pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.293077 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.292958 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/affdc3b9-a61c-4b68-9436-7f3846d48355-trusted-ca-bundle\") pod \"console-7cc68cc5c9-gs95r\" (UID: \"affdc3b9-a61c-4b68-9436-7f3846d48355\") " pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.293077 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.293050 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/affdc3b9-a61c-4b68-9436-7f3846d48355-console-config\") pod \"console-7cc68cc5c9-gs95r\" (UID: \"affdc3b9-a61c-4b68-9436-7f3846d48355\") " pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.293572 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.293086 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/affdc3b9-a61c-4b68-9436-7f3846d48355-console-serving-cert\") pod \"console-7cc68cc5c9-gs95r\" (UID: \"affdc3b9-a61c-4b68-9436-7f3846d48355\") " pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.293572 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.293108 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/affdc3b9-a61c-4b68-9436-7f3846d48355-console-oauth-config\") pod \"console-7cc68cc5c9-gs95r\" (UID: \"affdc3b9-a61c-4b68-9436-7f3846d48355\") " pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.293572 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.293134 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/affdc3b9-a61c-4b68-9436-7f3846d48355-oauth-serving-cert\") pod \"console-7cc68cc5c9-gs95r\" (UID: \"affdc3b9-a61c-4b68-9436-7f3846d48355\") " pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.293572 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.293158 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nt89\" (UniqueName: \"kubernetes.io/projected/affdc3b9-a61c-4b68-9436-7f3846d48355-kube-api-access-8nt89\") pod \"console-7cc68cc5c9-gs95r\" (UID: \"affdc3b9-a61c-4b68-9436-7f3846d48355\") " pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.293572 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.293431 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 17 17:56:10.298397 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.298378 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cc68cc5c9-gs95r"] Jul 17 17:56:10.354594 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.354566 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:10.393817 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.393783 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/affdc3b9-a61c-4b68-9436-7f3846d48355-console-config\") pod \"console-7cc68cc5c9-gs95r\" (UID: \"affdc3b9-a61c-4b68-9436-7f3846d48355\") " pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.393817 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.393820 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/affdc3b9-a61c-4b68-9436-7f3846d48355-console-serving-cert\") pod \"console-7cc68cc5c9-gs95r\" (UID: \"affdc3b9-a61c-4b68-9436-7f3846d48355\") " pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.394070 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.393835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/affdc3b9-a61c-4b68-9436-7f3846d48355-console-oauth-config\") pod \"console-7cc68cc5c9-gs95r\" (UID: \"affdc3b9-a61c-4b68-9436-7f3846d48355\") " pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.394070 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.393851 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/affdc3b9-a61c-4b68-9436-7f3846d48355-oauth-serving-cert\") pod \"console-7cc68cc5c9-gs95r\" (UID: \"affdc3b9-a61c-4b68-9436-7f3846d48355\") " pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.394070 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.393866 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nt89\" (UniqueName: \"kubernetes.io/projected/affdc3b9-a61c-4b68-9436-7f3846d48355-kube-api-access-8nt89\") pod \"console-7cc68cc5c9-gs95r\" (UID: \"affdc3b9-a61c-4b68-9436-7f3846d48355\") " pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.394070 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.393920 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/affdc3b9-a61c-4b68-9436-7f3846d48355-service-ca\") pod \"console-7cc68cc5c9-gs95r\" (UID: \"affdc3b9-a61c-4b68-9436-7f3846d48355\") " pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.394070 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.393941 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/affdc3b9-a61c-4b68-9436-7f3846d48355-trusted-ca-bundle\") pod \"console-7cc68cc5c9-gs95r\" (UID: \"affdc3b9-a61c-4b68-9436-7f3846d48355\") " pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.394625 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.394601 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/affdc3b9-a61c-4b68-9436-7f3846d48355-console-config\") pod \"console-7cc68cc5c9-gs95r\" (UID: \"affdc3b9-a61c-4b68-9436-7f3846d48355\") " pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.394800 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.394782 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/affdc3b9-a61c-4b68-9436-7f3846d48355-trusted-ca-bundle\") pod \"console-7cc68cc5c9-gs95r\" (UID: \"affdc3b9-a61c-4b68-9436-7f3846d48355\") " pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.394869 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.394848 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/affdc3b9-a61c-4b68-9436-7f3846d48355-oauth-serving-cert\") pod \"console-7cc68cc5c9-gs95r\" (UID: \"affdc3b9-a61c-4b68-9436-7f3846d48355\") " pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.394963 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.394943 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/affdc3b9-a61c-4b68-9436-7f3846d48355-service-ca\") pod \"console-7cc68cc5c9-gs95r\" (UID: \"affdc3b9-a61c-4b68-9436-7f3846d48355\") " pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.396540 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.396520 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/affdc3b9-a61c-4b68-9436-7f3846d48355-console-serving-cert\") pod \"console-7cc68cc5c9-gs95r\" (UID: \"affdc3b9-a61c-4b68-9436-7f3846d48355\") " pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.396632 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.396546 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/affdc3b9-a61c-4b68-9436-7f3846d48355-console-oauth-config\") pod \"console-7cc68cc5c9-gs95r\" (UID: \"affdc3b9-a61c-4b68-9436-7f3846d48355\") " pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.402073 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.402049 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nt89\" (UniqueName: \"kubernetes.io/projected/affdc3b9-a61c-4b68-9436-7f3846d48355-kube-api-access-8nt89\") pod \"console-7cc68cc5c9-gs95r\" (UID: \"affdc3b9-a61c-4b68-9436-7f3846d48355\") " pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.451099 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.451059 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:10.549239 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.549152 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:10.596237 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.596205 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:10.734227 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:10.734198 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cc68cc5c9-gs95r"] Jul 17 17:56:10.736759 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:56:10.736713 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaffdc3b9_a61c_4b68_9436_7f3846d48355.slice/crio-8d82be5665bf6f6cb28fc56c0e104febae6b24e0b1f24c5444b080c94a4f1ac2 WatchSource:0}: Error finding container 8d82be5665bf6f6cb28fc56c0e104febae6b24e0b1f24c5444b080c94a4f1ac2: Status 404 returned error can't find the container with id 8d82be5665bf6f6cb28fc56c0e104febae6b24e0b1f24c5444b080c94a4f1ac2 Jul 17 17:56:11.017519 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:11.017479 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cc68cc5c9-gs95r" event={"ID":"affdc3b9-a61c-4b68-9436-7f3846d48355","Type":"ContainerStarted","Data":"88af2eb6162e368a534bb2ccc6378d265f0e8fae4ff920f98d8ad177311a1df9"} Jul 17 17:56:11.017939 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:11.017541 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cc68cc5c9-gs95r" event={"ID":"affdc3b9-a61c-4b68-9436-7f3846d48355","Type":"ContainerStarted","Data":"8d82be5665bf6f6cb28fc56c0e104febae6b24e0b1f24c5444b080c94a4f1ac2"} Jul 17 17:56:11.035808 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:11.035757 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cc68cc5c9-gs95r" podStartSLOduration=1.03574003 podStartE2EDuration="1.03574003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:56:11.033837099 +0000 UTC m=+382.621789848" watchObservedRunningTime="2026-07-17 17:56:11.03574003 +0000 UTC m=+382.623692720" Jul 17 17:56:11.080982 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:11.080950 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:12.310884 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:12.310847 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:12.904464 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:12.904429 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:13.133757 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:13.133723 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:14.948112 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:14.948080 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:16.301930 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:16.301900 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:16.402623 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:16.402594 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:18.136807 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:18.136775 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:20.597406 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:20.597335 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:20.597803 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:20.597445 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:20.602579 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:20.602553 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:21.067061 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:21.067032 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cc68cc5c9-gs95r" Jul 17 17:56:21.219852 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:21.219824 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:21.320891 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:21.320798 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:21.420655 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:21.420620 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:21.522004 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:21.521974 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:24.703500 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:24.703445 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:24.803994 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:24.803957 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:29.513323 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:29.513284 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:29.885085 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:29.884996 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:30.510299 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:30.510268 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:30.569649 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:30.569621 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:30.757871 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:30.757838 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:30.857592 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:30.857513 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:33.859243 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:33.859202 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:33.958975 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:33.958934 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:39.855477 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:39.855442 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9jspx"] Jul 17 17:56:39.858613 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:39.858597 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9jspx" Jul 17 17:56:39.861314 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:39.861290 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 17:56:39.861474 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:39.861375 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 17:56:39.861474 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:39.861386 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hc8h4\"" Jul 17 17:56:39.869855 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:39.869832 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9jspx"] Jul 17 17:56:39.948705 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:39.948669 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2cb57bc9-7291-4034-9238-e46fefa2d764-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-9jspx\" (UID: \"2cb57bc9-7291-4034-9238-e46fefa2d764\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9jspx" Jul 17 17:56:39.948918 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:39.948716 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7282s\" (UniqueName: \"kubernetes.io/projected/2cb57bc9-7291-4034-9238-e46fefa2d764-kube-api-access-7282s\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-9jspx\" (UID: \"2cb57bc9-7291-4034-9238-e46fefa2d764\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9jspx" Jul 17 17:56:40.049643 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:40.049608 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2cb57bc9-7291-4034-9238-e46fefa2d764-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-9jspx\" (UID: \"2cb57bc9-7291-4034-9238-e46fefa2d764\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9jspx" Jul 17 17:56:40.049792 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:40.049652 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7282s\" (UniqueName: \"kubernetes.io/projected/2cb57bc9-7291-4034-9238-e46fefa2d764-kube-api-access-7282s\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-9jspx\" (UID: \"2cb57bc9-7291-4034-9238-e46fefa2d764\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9jspx" Jul 17 17:56:40.050039 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:40.050019 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2cb57bc9-7291-4034-9238-e46fefa2d764-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-9jspx\" (UID: \"2cb57bc9-7291-4034-9238-e46fefa2d764\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9jspx" Jul 17 17:56:40.057940 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:40.057916 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7282s\" (UniqueName: \"kubernetes.io/projected/2cb57bc9-7291-4034-9238-e46fefa2d764-kube-api-access-7282s\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-9jspx\" (UID: \"2cb57bc9-7291-4034-9238-e46fefa2d764\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9jspx" Jul 17 17:56:40.169785 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:40.169676 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9jspx" Jul 17 17:56:40.321001 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:40.320969 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9jspx"] Jul 17 17:56:40.324397 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:56:40.324342 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb57bc9_7291_4034_9238_e46fefa2d764.slice/crio-5b5370afcc6dfdd641bec1e69347b35c0f562edb0af28ab9678ca2d633741a54 WatchSource:0}: Error finding container 5b5370afcc6dfdd641bec1e69347b35c0f562edb0af28ab9678ca2d633741a54: Status 404 returned error can't find the container with id 5b5370afcc6dfdd641bec1e69347b35c0f562edb0af28ab9678ca2d633741a54 Jul 17 17:56:40.327469 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:40.327451 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 17:56:40.481917 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:40.481888 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:40.583460 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:40.583420 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:40.592801 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:40.592768 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9jspx"] Jul 17 17:56:40.601893 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:40.601863 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9jspx"] Jul 17 17:56:40.613871 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:40.613841 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8"] Jul 17 17:56:40.618471 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:40.618443 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8" Jul 17 17:56:40.630285 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:40.630255 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8"] Jul 17 17:56:40.756787 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:40.756683 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdc90b1e-bfae-4e41-8679-19f97bb6e0c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hbft8\" (UID: \"fdc90b1e-bfae-4e41-8679-19f97bb6e0c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8" Jul 17 17:56:40.756938 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:40.756861 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbv8q\" (UniqueName: \"kubernetes.io/projected/fdc90b1e-bfae-4e41-8679-19f97bb6e0c2-kube-api-access-vbv8q\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hbft8\" (UID: \"fdc90b1e-bfae-4e41-8679-19f97bb6e0c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8" Jul 17 17:56:40.858115 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:40.858080 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbv8q\" (UniqueName: \"kubernetes.io/projected/fdc90b1e-bfae-4e41-8679-19f97bb6e0c2-kube-api-access-vbv8q\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hbft8\" (UID: \"fdc90b1e-bfae-4e41-8679-19f97bb6e0c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8" Jul 17 17:56:40.858569 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:40.858141 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdc90b1e-bfae-4e41-8679-19f97bb6e0c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hbft8\" (UID: \"fdc90b1e-bfae-4e41-8679-19f97bb6e0c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8" Jul 17 17:56:40.858569 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:40.858519 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdc90b1e-bfae-4e41-8679-19f97bb6e0c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hbft8\" (UID: \"fdc90b1e-bfae-4e41-8679-19f97bb6e0c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8" Jul 17 17:56:40.866604 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:40.866579 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbv8q\" (UniqueName: \"kubernetes.io/projected/fdc90b1e-bfae-4e41-8679-19f97bb6e0c2-kube-api-access-vbv8q\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hbft8\" (UID: \"fdc90b1e-bfae-4e41-8679-19f97bb6e0c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8" Jul 17 17:56:40.931211 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:40.931179 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8" Jul 17 17:56:41.278775 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:41.278747 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8"] Jul 17 17:56:41.326204 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:56:41.326162 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc90b1e_bfae_4e41_8679_19f97bb6e0c2.slice/crio-59c227397bcf06019a3c223be39393c5070796083e7bfec973f6701ea4a7e3b2 WatchSource:0}: Error finding container 59c227397bcf06019a3c223be39393c5070796083e7bfec973f6701ea4a7e3b2: Status 404 returned error can't find the container with id 59c227397bcf06019a3c223be39393c5070796083e7bfec973f6701ea4a7e3b2 Jul 17 17:56:42.167090 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:42.167049 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8" event={"ID":"fdc90b1e-bfae-4e41-8679-19f97bb6e0c2","Type":"ContainerStarted","Data":"59c227397bcf06019a3c223be39393c5070796083e7bfec973f6701ea4a7e3b2"} Jul 17 17:56:42.648450 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:42.648339 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:42.747405 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:42.747371 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:43.668550 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:43.668512 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:43.727892 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:43.727856 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:46.188486 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:46.188444 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8" event={"ID":"fdc90b1e-bfae-4e41-8679-19f97bb6e0c2","Type":"ContainerStarted","Data":"bcc685e0aa5a0bbef2f969927cb8f6e4f8bd02f6f07a0f0c047fbf312e3fb9c7"} Jul 17 17:56:46.188975 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:46.188641 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8" Jul 17 17:56:46.190270 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:46.190241 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9jspx" podUID="2cb57bc9-7291-4034-9238-e46fefa2d764" containerName="manager" containerID="cri-o://cfec525ef5f4b81d3fb3b33429384ba2cae4790a4ecb8a0b12575dba85830ba2" gracePeriod=2 Jul 17 17:56:46.209689 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:46.209637 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8" podStartSLOduration=2.390332383 podStartE2EDuration="6.209620432s" totalImagesPullingTime="3.819288049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:56:46.207408462 +0000 UTC m=+417.795361172" watchObservedRunningTime="2026-07-17 17:56:46.209620432 +0000 UTC m=+417.797573113" Jul 17 17:56:46.433394 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:46.433368 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9jspx" Jul 17 17:56:46.435716 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:56:46.435690 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-9jspx\" is forbidden: User \"system:node:ip-10-0-142-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-155.ec2.internal' and this object" podUID="2cb57bc9-7291-4034-9238-e46fefa2d764" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9jspx" Jul 17 17:56:46.516027 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:46.515991 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2cb57bc9-7291-4034-9238-e46fefa2d764-extensions-socket-volume\") pod \"2cb57bc9-7291-4034-9238-e46fefa2d764\" (UID: \"2cb57bc9-7291-4034-9238-e46fefa2d764\") " Jul 17 17:56:46.516209 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:46.516087 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7282s\" (UniqueName: \"kubernetes.io/projected/2cb57bc9-7291-4034-9238-e46fefa2d764-kube-api-access-7282s\") pod \"2cb57bc9-7291-4034-9238-e46fefa2d764\" (UID: \"2cb57bc9-7291-4034-9238-e46fefa2d764\") " Jul 17 17:56:46.516290 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:46.516263 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb57bc9-7291-4034-9238-e46fefa2d764-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2cb57bc9-7291-4034-9238-e46fefa2d764" (UID: "2cb57bc9-7291-4034-9238-e46fefa2d764"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:56:46.518515 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:46.518488 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb57bc9-7291-4034-9238-e46fefa2d764-kube-api-access-7282s" (OuterVolumeSpecName: "kube-api-access-7282s") pod "2cb57bc9-7291-4034-9238-e46fefa2d764" (UID: "2cb57bc9-7291-4034-9238-e46fefa2d764"). InnerVolumeSpecName "kube-api-access-7282s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:56:46.617616 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:46.617572 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7282s\" (UniqueName: \"kubernetes.io/projected/2cb57bc9-7291-4034-9238-e46fefa2d764-kube-api-access-7282s\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:56:46.617616 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:46.617607 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2cb57bc9-7291-4034-9238-e46fefa2d764-extensions-socket-volume\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:56:46.985468 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:46.985425 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cb57bc9-7291-4034-9238-e46fefa2d764" path="/var/lib/kubelet/pods/2cb57bc9-7291-4034-9238-e46fefa2d764/volumes" Jul 17 17:56:47.196431 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:47.196400 2574 generic.go:356] "Generic (PLEG): container finished" podID="2cb57bc9-7291-4034-9238-e46fefa2d764" containerID="cfec525ef5f4b81d3fb3b33429384ba2cae4790a4ecb8a0b12575dba85830ba2" exitCode=2 Jul 17 17:56:47.196884 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:47.196451 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9jspx" Jul 17 17:56:47.196884 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:47.196504 2574 scope.go:117] "RemoveContainer" containerID="cfec525ef5f4b81d3fb3b33429384ba2cae4790a4ecb8a0b12575dba85830ba2" Jul 17 17:56:47.201248 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:56:47.201217 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-9jspx\" is forbidden: User \"system:node:ip-10-0-142-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-155.ec2.internal' and this object" podUID="2cb57bc9-7291-4034-9238-e46fefa2d764" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9jspx" Jul 17 17:56:47.205472 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:47.205447 2574 scope.go:117] "RemoveContainer" containerID="cfec525ef5f4b81d3fb3b33429384ba2cae4790a4ecb8a0b12575dba85830ba2" Jul 17 17:56:47.205786 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:56:47.205765 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfec525ef5f4b81d3fb3b33429384ba2cae4790a4ecb8a0b12575dba85830ba2\": container with ID starting with cfec525ef5f4b81d3fb3b33429384ba2cae4790a4ecb8a0b12575dba85830ba2 not found: ID does not exist" containerID="cfec525ef5f4b81d3fb3b33429384ba2cae4790a4ecb8a0b12575dba85830ba2" Jul 17 17:56:47.205867 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:47.205798 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfec525ef5f4b81d3fb3b33429384ba2cae4790a4ecb8a0b12575dba85830ba2"} err="failed to get container status \"cfec525ef5f4b81d3fb3b33429384ba2cae4790a4ecb8a0b12575dba85830ba2\": rpc error: code = NotFound desc = could not find container \"cfec525ef5f4b81d3fb3b33429384ba2cae4790a4ecb8a0b12575dba85830ba2\": container with ID starting with cfec525ef5f4b81d3fb3b33429384ba2cae4790a4ecb8a0b12575dba85830ba2 not found: ID does not exist" Jul 17 17:56:48.984655 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:56:48.984608 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-9jspx\" is forbidden: User \"system:node:ip-10-0-142-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-155.ec2.internal' and this object" podUID="2cb57bc9-7291-4034-9238-e46fefa2d764" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-9jspx" Jul 17 17:56:49.668687 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:49.668656 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:49.719531 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:49.719495 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:49.868287 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:49.868256 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:49.967509 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:49.967480 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:52.609265 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:52.609226 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:52.709706 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:52.709672 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:56:57.199030 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:57.198993 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8" Jul 17 17:56:57.510670 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:57.510631 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td"] Jul 17 17:56:57.511017 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:57.511004 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cb57bc9-7291-4034-9238-e46fefa2d764" containerName="manager" Jul 17 17:56:57.511059 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:57.511018 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cb57bc9-7291-4034-9238-e46fefa2d764" containerName="manager" Jul 17 17:56:57.511093 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:57.511087 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cb57bc9-7291-4034-9238-e46fefa2d764" containerName="manager" Jul 17 17:56:57.515697 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:57.515676 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td" Jul 17 17:56:57.531733 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:57.531697 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td"] Jul 17 17:56:57.618229 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:57.618184 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1cd3e533-bd47-43dc-b45d-32f3ff4827df-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-pl6td\" (UID: \"1cd3e533-bd47-43dc-b45d-32f3ff4827df\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td" Jul 17 17:56:57.618442 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:57.618299 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bbfd\" (UniqueName: \"kubernetes.io/projected/1cd3e533-bd47-43dc-b45d-32f3ff4827df-kube-api-access-8bbfd\") pod \"kuadrant-operator-controller-manager-86878c769f-pl6td\" (UID: \"1cd3e533-bd47-43dc-b45d-32f3ff4827df\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td" Jul 17 17:56:57.719003 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:57.718962 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1cd3e533-bd47-43dc-b45d-32f3ff4827df-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-pl6td\" (UID: \"1cd3e533-bd47-43dc-b45d-32f3ff4827df\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td" Jul 17 17:56:57.719176 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:57.719040 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbfd\" (UniqueName: \"kubernetes.io/projected/1cd3e533-bd47-43dc-b45d-32f3ff4827df-kube-api-access-8bbfd\") pod \"kuadrant-operator-controller-manager-86878c769f-pl6td\" (UID: \"1cd3e533-bd47-43dc-b45d-32f3ff4827df\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td" Jul 17 17:56:57.719396 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:57.719375 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1cd3e533-bd47-43dc-b45d-32f3ff4827df-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-pl6td\" (UID: \"1cd3e533-bd47-43dc-b45d-32f3ff4827df\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td" Jul 17 17:56:57.727812 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:57.727745 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bbfd\" (UniqueName: \"kubernetes.io/projected/1cd3e533-bd47-43dc-b45d-32f3ff4827df-kube-api-access-8bbfd\") pod \"kuadrant-operator-controller-manager-86878c769f-pl6td\" (UID: \"1cd3e533-bd47-43dc-b45d-32f3ff4827df\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td" Jul 17 17:56:57.825855 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:57.825808 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td" Jul 17 17:56:57.967313 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:57.967288 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td"] Jul 17 17:56:57.969708 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:56:57.969686 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd3e533_bd47_43dc_b45d_32f3ff4827df.slice/crio-8ad34a1e695b138778b5d22ec204a3e809719050d8367c8fac0a7152316bed3b WatchSource:0}: Error finding container 8ad34a1e695b138778b5d22ec204a3e809719050d8367c8fac0a7152316bed3b: Status 404 returned error can't find the container with id 8ad34a1e695b138778b5d22ec204a3e809719050d8367c8fac0a7152316bed3b Jul 17 17:56:58.251794 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:58.251698 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td" event={"ID":"1cd3e533-bd47-43dc-b45d-32f3ff4827df","Type":"ContainerStarted","Data":"264c3f8efa5e44af5e3264b0062b5f16cf265e1287984cb88939838c31c72e99"} Jul 17 17:56:58.251794 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:58.251758 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td" event={"ID":"1cd3e533-bd47-43dc-b45d-32f3ff4827df","Type":"ContainerStarted","Data":"8ad34a1e695b138778b5d22ec204a3e809719050d8367c8fac0a7152316bed3b"} Jul 17 17:56:58.252191 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:58.251897 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td" Jul 17 17:56:58.280915 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:56:58.280848 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td" podStartSLOduration=1.280831807 podStartE2EDuration="1.280831807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:56:58.280186277 +0000 UTC m=+429.868138973" watchObservedRunningTime="2026-07-17 17:56:58.280831807 +0000 UTC m=+429.868784496" Jul 17 17:57:08.718841 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:08.718807 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:08.776474 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:08.776449 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:09.258506 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:09.258475 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td" Jul 17 17:57:09.308181 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:09.308148 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8"] Jul 17 17:57:09.308447 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:09.308416 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8" podUID="fdc90b1e-bfae-4e41-8679-19f97bb6e0c2" containerName="manager" containerID="cri-o://bcc685e0aa5a0bbef2f969927cb8f6e4f8bd02f6f07a0f0c047fbf312e3fb9c7" gracePeriod=10 Jul 17 17:57:09.558106 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:09.558081 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8" Jul 17 17:57:09.619951 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:09.619923 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbv8q\" (UniqueName: \"kubernetes.io/projected/fdc90b1e-bfae-4e41-8679-19f97bb6e0c2-kube-api-access-vbv8q\") pod \"fdc90b1e-bfae-4e41-8679-19f97bb6e0c2\" (UID: \"fdc90b1e-bfae-4e41-8679-19f97bb6e0c2\") " Jul 17 17:57:09.620080 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:09.619975 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdc90b1e-bfae-4e41-8679-19f97bb6e0c2-extensions-socket-volume\") pod \"fdc90b1e-bfae-4e41-8679-19f97bb6e0c2\" (UID: \"fdc90b1e-bfae-4e41-8679-19f97bb6e0c2\") " Jul 17 17:57:09.620364 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:09.620332 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc90b1e-bfae-4e41-8679-19f97bb6e0c2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fdc90b1e-bfae-4e41-8679-19f97bb6e0c2" (UID: "fdc90b1e-bfae-4e41-8679-19f97bb6e0c2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 17:57:09.622029 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:09.622007 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc90b1e-bfae-4e41-8679-19f97bb6e0c2-kube-api-access-vbv8q" (OuterVolumeSpecName: "kube-api-access-vbv8q") pod "fdc90b1e-bfae-4e41-8679-19f97bb6e0c2" (UID: "fdc90b1e-bfae-4e41-8679-19f97bb6e0c2"). InnerVolumeSpecName "kube-api-access-vbv8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:57:09.721662 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:09.721626 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbv8q\" (UniqueName: \"kubernetes.io/projected/fdc90b1e-bfae-4e41-8679-19f97bb6e0c2-kube-api-access-vbv8q\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:57:09.721662 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:09.721660 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdc90b1e-bfae-4e41-8679-19f97bb6e0c2-extensions-socket-volume\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:57:10.309375 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:10.309330 2574 generic.go:356] "Generic (PLEG): container finished" podID="fdc90b1e-bfae-4e41-8679-19f97bb6e0c2" containerID="bcc685e0aa5a0bbef2f969927cb8f6e4f8bd02f6f07a0f0c047fbf312e3fb9c7" exitCode=0 Jul 17 17:57:10.309541 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:10.309402 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8" Jul 17 17:57:10.309541 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:10.309422 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8" event={"ID":"fdc90b1e-bfae-4e41-8679-19f97bb6e0c2","Type":"ContainerDied","Data":"bcc685e0aa5a0bbef2f969927cb8f6e4f8bd02f6f07a0f0c047fbf312e3fb9c7"} Jul 17 17:57:10.309541 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:10.309481 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8" event={"ID":"fdc90b1e-bfae-4e41-8679-19f97bb6e0c2","Type":"ContainerDied","Data":"59c227397bcf06019a3c223be39393c5070796083e7bfec973f6701ea4a7e3b2"} Jul 17 17:57:10.309541 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:10.309511 2574 scope.go:117] "RemoveContainer" containerID="bcc685e0aa5a0bbef2f969927cb8f6e4f8bd02f6f07a0f0c047fbf312e3fb9c7" Jul 17 17:57:10.318878 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:10.318859 2574 scope.go:117] "RemoveContainer" containerID="bcc685e0aa5a0bbef2f969927cb8f6e4f8bd02f6f07a0f0c047fbf312e3fb9c7" Jul 17 17:57:10.319140 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:57:10.319118 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcc685e0aa5a0bbef2f969927cb8f6e4f8bd02f6f07a0f0c047fbf312e3fb9c7\": container with ID starting with bcc685e0aa5a0bbef2f969927cb8f6e4f8bd02f6f07a0f0c047fbf312e3fb9c7 not found: ID does not exist" containerID="bcc685e0aa5a0bbef2f969927cb8f6e4f8bd02f6f07a0f0c047fbf312e3fb9c7" Jul 17 17:57:10.319231 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:10.319144 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcc685e0aa5a0bbef2f969927cb8f6e4f8bd02f6f07a0f0c047fbf312e3fb9c7"} err="failed to get container status \"bcc685e0aa5a0bbef2f969927cb8f6e4f8bd02f6f07a0f0c047fbf312e3fb9c7\": rpc error: code = NotFound desc = could not find container \"bcc685e0aa5a0bbef2f969927cb8f6e4f8bd02f6f07a0f0c047fbf312e3fb9c7\": container with ID starting with bcc685e0aa5a0bbef2f969927cb8f6e4f8bd02f6f07a0f0c047fbf312e3fb9c7 not found: ID does not exist" Jul 17 17:57:10.330957 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:10.330936 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8"] Jul 17 17:57:10.334428 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:10.334409 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hbft8"] Jul 17 17:57:10.984553 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:10.984521 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdc90b1e-bfae-4e41-8679-19f97bb6e0c2" path="/var/lib/kubelet/pods/fdc90b1e-bfae-4e41-8679-19f97bb6e0c2/volumes" Jul 17 17:57:13.704304 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:13.704266 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:14.087788 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:14.087751 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:17.101378 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:17.101317 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:17.202134 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:17.202099 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:27.339618 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:27.339587 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:27.451655 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:27.451625 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:27.509320 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:27.509288 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:28.134316 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:28.134281 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-d2554"] Jul 17 17:57:28.134679 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:28.134667 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdc90b1e-bfae-4e41-8679-19f97bb6e0c2" containerName="manager" Jul 17 17:57:28.134726 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:28.134681 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdc90b1e-bfae-4e41-8679-19f97bb6e0c2" containerName="manager" Jul 17 17:57:28.134761 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:28.134744 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdc90b1e-bfae-4e41-8679-19f97bb6e0c2" containerName="manager" Jul 17 17:57:28.140566 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:28.140542 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-d2554" Jul 17 17:57:28.141957 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:28.141933 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-d2554"] Jul 17 17:57:28.143286 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:28.143258 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hp79g\"" Jul 17 17:57:28.166877 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:28.166837 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:28.225923 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:28.225891 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:28.294296 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:28.294262 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqgrw\" (UniqueName: \"kubernetes.io/projected/7357f219-31de-449f-9517-e1d76b9fc716-kube-api-access-dqgrw\") pod \"authorino-58cd955675-d2554\" (UID: \"7357f219-31de-449f-9517-e1d76b9fc716\") " pod="kuadrant-system/authorino-58cd955675-d2554" Jul 17 17:57:28.395861 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:28.395769 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqgrw\" (UniqueName: \"kubernetes.io/projected/7357f219-31de-449f-9517-e1d76b9fc716-kube-api-access-dqgrw\") pod \"authorino-58cd955675-d2554\" (UID: \"7357f219-31de-449f-9517-e1d76b9fc716\") " pod="kuadrant-system/authorino-58cd955675-d2554" Jul 17 17:57:28.406254 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:28.406224 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqgrw\" (UniqueName: \"kubernetes.io/projected/7357f219-31de-449f-9517-e1d76b9fc716-kube-api-access-dqgrw\") pod \"authorino-58cd955675-d2554\" (UID: \"7357f219-31de-449f-9517-e1d76b9fc716\") " pod="kuadrant-system/authorino-58cd955675-d2554" Jul 17 17:57:28.420820 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:28.420790 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-rlqv9"] Jul 17 17:57:28.424100 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:28.424085 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-rlqv9" Jul 17 17:57:28.430378 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:28.430338 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-rlqv9"] Jul 17 17:57:28.452369 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:28.452329 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-d2554" Jul 17 17:57:28.496791 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:28.496751 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt2gt\" (UniqueName: \"kubernetes.io/projected/87941017-722f-4f9b-9f2b-33718a2ee9ae-kube-api-access-pt2gt\") pod \"authorino-75d556bd77-rlqv9\" (UID: \"87941017-722f-4f9b-9f2b-33718a2ee9ae\") " pod="kuadrant-system/authorino-75d556bd77-rlqv9" Jul 17 17:57:28.587010 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:28.586975 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-d2554"] Jul 17 17:57:28.590504 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:57:28.590475 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7357f219_31de_449f_9517_e1d76b9fc716.slice/crio-54963e1f0538177f515fc1b9ad9378f9e28a9d19d34fd0a64e659e82a02c496c WatchSource:0}: Error finding container 54963e1f0538177f515fc1b9ad9378f9e28a9d19d34fd0a64e659e82a02c496c: Status 404 returned error can't find the container with id 54963e1f0538177f515fc1b9ad9378f9e28a9d19d34fd0a64e659e82a02c496c Jul 17 17:57:28.597989 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:28.597964 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt2gt\" (UniqueName: \"kubernetes.io/projected/87941017-722f-4f9b-9f2b-33718a2ee9ae-kube-api-access-pt2gt\") pod \"authorino-75d556bd77-rlqv9\" (UID: \"87941017-722f-4f9b-9f2b-33718a2ee9ae\") " pod="kuadrant-system/authorino-75d556bd77-rlqv9" Jul 17 17:57:28.607106 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:28.607078 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt2gt\" (UniqueName: \"kubernetes.io/projected/87941017-722f-4f9b-9f2b-33718a2ee9ae-kube-api-access-pt2gt\") pod \"authorino-75d556bd77-rlqv9\" (UID: \"87941017-722f-4f9b-9f2b-33718a2ee9ae\") " pod="kuadrant-system/authorino-75d556bd77-rlqv9" Jul 17 17:57:28.735144 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:28.735110 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-rlqv9" Jul 17 17:57:28.870797 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:28.870771 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-rlqv9"] Jul 17 17:57:28.873470 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:57:28.873439 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87941017_722f_4f9b_9f2b_33718a2ee9ae.slice/crio-1172a91bdfa34d388deb8df65144bdc979040ffeb5e08b58c19aa519a01fe63c WatchSource:0}: Error finding container 1172a91bdfa34d388deb8df65144bdc979040ffeb5e08b58c19aa519a01fe63c: Status 404 returned error can't find the container with id 1172a91bdfa34d388deb8df65144bdc979040ffeb5e08b58c19aa519a01fe63c Jul 17 17:57:29.412501 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:29.412461 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-rlqv9" event={"ID":"87941017-722f-4f9b-9f2b-33718a2ee9ae","Type":"ContainerStarted","Data":"1172a91bdfa34d388deb8df65144bdc979040ffeb5e08b58c19aa519a01fe63c"} Jul 17 17:57:29.416663 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:29.416627 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-d2554" event={"ID":"7357f219-31de-449f-9517-e1d76b9fc716","Type":"ContainerStarted","Data":"54963e1f0538177f515fc1b9ad9378f9e28a9d19d34fd0a64e659e82a02c496c"} Jul 17 17:57:32.436245 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:32.436203 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-rlqv9" event={"ID":"87941017-722f-4f9b-9f2b-33718a2ee9ae","Type":"ContainerStarted","Data":"dd9d290ad5d4394dee3e832de079b152f69ee437fa327aa44c48882be3594e2d"} Jul 17 17:57:32.437610 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:32.437581 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-d2554" event={"ID":"7357f219-31de-449f-9517-e1d76b9fc716","Type":"ContainerStarted","Data":"cf7a7bb0a56b49f4b698bbc9ebb6c163bf5c4d3854d102cfdf33721c67e5af96"} Jul 17 17:57:32.450340 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:32.450288 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-rlqv9" podStartSLOduration=1.070544928 podStartE2EDuration="4.45027148s" totalImagesPullingTime="3.379726552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:57:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:57:32.449520822 +0000 UTC m=+464.037473515" watchObservedRunningTime="2026-07-17 17:57:32.45027148 +0000 UTC m=+464.038224170" Jul 17 17:57:32.465994 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:32.465930 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-d2554" podStartSLOduration=0.814726616 podStartE2EDuration="4.465906954s" totalImagesPullingTime="3.651180338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:57:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:57:32.465812695 +0000 UTC m=+464.053765386" watchObservedRunningTime="2026-07-17 17:57:32.465906954 +0000 UTC m=+464.053859645" Jul 17 17:57:32.475734 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:32.475706 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-d2554"] Jul 17 17:57:34.447937 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:34.447897 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-d2554" podUID="7357f219-31de-449f-9517-e1d76b9fc716" containerName="authorino" containerID="cri-o://cf7a7bb0a56b49f4b698bbc9ebb6c163bf5c4d3854d102cfdf33721c67e5af96" gracePeriod=30 Jul 17 17:57:34.714248 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:34.714224 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-d2554" Jul 17 17:57:34.860525 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:34.860485 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqgrw\" (UniqueName: \"kubernetes.io/projected/7357f219-31de-449f-9517-e1d76b9fc716-kube-api-access-dqgrw\") pod \"7357f219-31de-449f-9517-e1d76b9fc716\" (UID: \"7357f219-31de-449f-9517-e1d76b9fc716\") " Jul 17 17:57:34.862830 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:34.862806 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7357f219-31de-449f-9517-e1d76b9fc716-kube-api-access-dqgrw" (OuterVolumeSpecName: "kube-api-access-dqgrw") pod "7357f219-31de-449f-9517-e1d76b9fc716" (UID: "7357f219-31de-449f-9517-e1d76b9fc716"). InnerVolumeSpecName "kube-api-access-dqgrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:57:34.961536 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:34.961456 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqgrw\" (UniqueName: \"kubernetes.io/projected/7357f219-31de-449f-9517-e1d76b9fc716-kube-api-access-dqgrw\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:57:35.454281 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:35.454246 2574 generic.go:356] "Generic (PLEG): container finished" podID="7357f219-31de-449f-9517-e1d76b9fc716" containerID="cf7a7bb0a56b49f4b698bbc9ebb6c163bf5c4d3854d102cfdf33721c67e5af96" exitCode=0 Jul 17 17:57:35.454789 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:35.454282 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-d2554" event={"ID":"7357f219-31de-449f-9517-e1d76b9fc716","Type":"ContainerDied","Data":"cf7a7bb0a56b49f4b698bbc9ebb6c163bf5c4d3854d102cfdf33721c67e5af96"} Jul 17 17:57:35.454789 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:35.454333 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-d2554" event={"ID":"7357f219-31de-449f-9517-e1d76b9fc716","Type":"ContainerDied","Data":"54963e1f0538177f515fc1b9ad9378f9e28a9d19d34fd0a64e659e82a02c496c"} Jul 17 17:57:35.454789 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:35.454292 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-d2554" Jul 17 17:57:35.454789 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:35.454349 2574 scope.go:117] "RemoveContainer" containerID="cf7a7bb0a56b49f4b698bbc9ebb6c163bf5c4d3854d102cfdf33721c67e5af96" Jul 17 17:57:35.463223 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:35.463203 2574 scope.go:117] "RemoveContainer" containerID="cf7a7bb0a56b49f4b698bbc9ebb6c163bf5c4d3854d102cfdf33721c67e5af96" Jul 17 17:57:35.463508 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:57:35.463489 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf7a7bb0a56b49f4b698bbc9ebb6c163bf5c4d3854d102cfdf33721c67e5af96\": container with ID starting with cf7a7bb0a56b49f4b698bbc9ebb6c163bf5c4d3854d102cfdf33721c67e5af96 not found: ID does not exist" containerID="cf7a7bb0a56b49f4b698bbc9ebb6c163bf5c4d3854d102cfdf33721c67e5af96" Jul 17 17:57:35.463573 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:35.463517 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf7a7bb0a56b49f4b698bbc9ebb6c163bf5c4d3854d102cfdf33721c67e5af96"} err="failed to get container status \"cf7a7bb0a56b49f4b698bbc9ebb6c163bf5c4d3854d102cfdf33721c67e5af96\": rpc error: code = NotFound desc = could not find container \"cf7a7bb0a56b49f4b698bbc9ebb6c163bf5c4d3854d102cfdf33721c67e5af96\": container with ID starting with cf7a7bb0a56b49f4b698bbc9ebb6c163bf5c4d3854d102cfdf33721c67e5af96 not found: ID does not exist" Jul 17 17:57:35.471713 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:35.471691 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-d2554"] Jul 17 17:57:35.475565 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:35.475541 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-d2554"] Jul 17 17:57:36.986324 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:36.986284 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7357f219-31de-449f-9517-e1d76b9fc716" path="/var/lib/kubelet/pods/7357f219-31de-449f-9517-e1d76b9fc716/volumes" Jul 17 17:57:47.716958 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:47.716901 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:47.822644 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:47.822609 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:57:48.227762 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:57:48.227728 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:58:00.385484 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:00.385449 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:58:00.628853 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:00.628817 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-r9p9d"] Jul 17 17:58:00.629400 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:00.629382 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7357f219-31de-449f-9517-e1d76b9fc716" containerName="authorino" Jul 17 17:58:00.629461 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:00.629403 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7357f219-31de-449f-9517-e1d76b9fc716" containerName="authorino" Jul 17 17:58:00.629501 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:00.629467 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7357f219-31de-449f-9517-e1d76b9fc716" containerName="authorino" Jul 17 17:58:00.633809 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:00.633789 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-r9p9d" Jul 17 17:58:00.638214 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:00.638157 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-r9p9d"] Jul 17 17:58:00.680251 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:00.680218 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:58:00.703547 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:00.703511 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffvkt\" (UniqueName: \"kubernetes.io/projected/2cfde866-95bf-4f90-b8e1-15661fba6b87-kube-api-access-ffvkt\") pod \"authorino-5d4b67c446-r9p9d\" (UID: \"2cfde866-95bf-4f90-b8e1-15661fba6b87\") " pod="kuadrant-system/authorino-5d4b67c446-r9p9d" Jul 17 17:58:00.804179 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:00.804142 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffvkt\" (UniqueName: \"kubernetes.io/projected/2cfde866-95bf-4f90-b8e1-15661fba6b87-kube-api-access-ffvkt\") pod \"authorino-5d4b67c446-r9p9d\" (UID: \"2cfde866-95bf-4f90-b8e1-15661fba6b87\") " pod="kuadrant-system/authorino-5d4b67c446-r9p9d" Jul 17 17:58:00.812201 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:00.812165 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffvkt\" (UniqueName: \"kubernetes.io/projected/2cfde866-95bf-4f90-b8e1-15661fba6b87-kube-api-access-ffvkt\") pod \"authorino-5d4b67c446-r9p9d\" (UID: \"2cfde866-95bf-4f90-b8e1-15661fba6b87\") " pod="kuadrant-system/authorino-5d4b67c446-r9p9d" Jul 17 17:58:00.860604 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:00.860565 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-r9p9d"] Jul 17 17:58:00.860889 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:00.860873 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-r9p9d" Jul 17 17:58:00.885937 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:00.885904 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-575c877f5f-4jvwf"] Jul 17 17:58:00.892257 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:00.892193 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-575c877f5f-4jvwf" Jul 17 17:58:00.895265 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:00.895242 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-575c877f5f-4jvwf"] Jul 17 17:58:00.970892 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:00.970860 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-575c877f5f-4jvwf"] Jul 17 17:58:00.971341 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:58:00.971306 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xsnsk], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-575c877f5f-4jvwf" podUID="c836651f-7405-496d-b414-afe9f0adc7e5" Jul 17 17:58:00.996227 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:00.996195 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6cf6cf6459-xpt2h"] Jul 17 17:58:00.999663 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:00.999646 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cf6cf6459-xpt2h" Jul 17 17:58:01.002287 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.002248 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 17 17:58:01.006171 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.006140 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsnsk\" (UniqueName: \"kubernetes.io/projected/c836651f-7405-496d-b414-afe9f0adc7e5-kube-api-access-xsnsk\") pod \"authorino-575c877f5f-4jvwf\" (UID: \"c836651f-7405-496d-b414-afe9f0adc7e5\") " pod="kuadrant-system/authorino-575c877f5f-4jvwf" Jul 17 17:58:01.006935 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.006171 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cf6cf6459-xpt2h"] Jul 17 17:58:01.010139 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.010119 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-r9p9d"] Jul 17 17:58:01.012633 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:58:01.012609 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cfde866_95bf_4f90_b8e1_15661fba6b87.slice/crio-8be6a8639194525b8b2870bab66939853776d42c3034d1ad73439e7d89b6dcbe WatchSource:0}: Error finding container 8be6a8639194525b8b2870bab66939853776d42c3034d1ad73439e7d89b6dcbe: Status 404 returned error can't find the container with id 8be6a8639194525b8b2870bab66939853776d42c3034d1ad73439e7d89b6dcbe Jul 17 17:58:01.108565 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.108516 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ce07bba1-873f-42d5-ad2a-464e597edd18-tls-cert\") pod \"authorino-6cf6cf6459-xpt2h\" (UID: \"ce07bba1-873f-42d5-ad2a-464e597edd18\") " pod="kuadrant-system/authorino-6cf6cf6459-xpt2h" Jul 17 17:58:01.108781 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.108576 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts79k\" (UniqueName: \"kubernetes.io/projected/ce07bba1-873f-42d5-ad2a-464e597edd18-kube-api-access-ts79k\") pod \"authorino-6cf6cf6459-xpt2h\" (UID: \"ce07bba1-873f-42d5-ad2a-464e597edd18\") " pod="kuadrant-system/authorino-6cf6cf6459-xpt2h" Jul 17 17:58:01.108781 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.108697 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsnsk\" (UniqueName: \"kubernetes.io/projected/c836651f-7405-496d-b414-afe9f0adc7e5-kube-api-access-xsnsk\") pod \"authorino-575c877f5f-4jvwf\" (UID: \"c836651f-7405-496d-b414-afe9f0adc7e5\") " pod="kuadrant-system/authorino-575c877f5f-4jvwf" Jul 17 17:58:01.116434 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.116409 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsnsk\" (UniqueName: \"kubernetes.io/projected/c836651f-7405-496d-b414-afe9f0adc7e5-kube-api-access-xsnsk\") pod \"authorino-575c877f5f-4jvwf\" (UID: \"c836651f-7405-496d-b414-afe9f0adc7e5\") " pod="kuadrant-system/authorino-575c877f5f-4jvwf" Jul 17 17:58:01.209741 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.209647 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ce07bba1-873f-42d5-ad2a-464e597edd18-tls-cert\") pod \"authorino-6cf6cf6459-xpt2h\" (UID: \"ce07bba1-873f-42d5-ad2a-464e597edd18\") " pod="kuadrant-system/authorino-6cf6cf6459-xpt2h" Jul 17 17:58:01.209741 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.209686 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts79k\" (UniqueName: \"kubernetes.io/projected/ce07bba1-873f-42d5-ad2a-464e597edd18-kube-api-access-ts79k\") pod \"authorino-6cf6cf6459-xpt2h\" (UID: \"ce07bba1-873f-42d5-ad2a-464e597edd18\") " pod="kuadrant-system/authorino-6cf6cf6459-xpt2h" Jul 17 17:58:01.212329 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.212305 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ce07bba1-873f-42d5-ad2a-464e597edd18-tls-cert\") pod \"authorino-6cf6cf6459-xpt2h\" (UID: \"ce07bba1-873f-42d5-ad2a-464e597edd18\") " pod="kuadrant-system/authorino-6cf6cf6459-xpt2h" Jul 17 17:58:01.217261 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.217236 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts79k\" (UniqueName: \"kubernetes.io/projected/ce07bba1-873f-42d5-ad2a-464e597edd18-kube-api-access-ts79k\") pod \"authorino-6cf6cf6459-xpt2h\" (UID: \"ce07bba1-873f-42d5-ad2a-464e597edd18\") " pod="kuadrant-system/authorino-6cf6cf6459-xpt2h" Jul 17 17:58:01.310588 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.310551 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cf6cf6459-xpt2h" Jul 17 17:58:01.447000 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.446973 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cf6cf6459-xpt2h"] Jul 17 17:58:01.448778 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:58:01.448753 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce07bba1_873f_42d5_ad2a_464e597edd18.slice/crio-4f6a83614c213847958671023e2c4ef7f0be2a10deb4a4e81d582a054ffd317a WatchSource:0}: Error finding container 4f6a83614c213847958671023e2c4ef7f0be2a10deb4a4e81d582a054ffd317a: Status 404 returned error can't find the container with id 4f6a83614c213847958671023e2c4ef7f0be2a10deb4a4e81d582a054ffd317a Jul 17 17:58:01.580883 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.580842 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cf6cf6459-xpt2h" event={"ID":"ce07bba1-873f-42d5-ad2a-464e597edd18","Type":"ContainerStarted","Data":"4f6a83614c213847958671023e2c4ef7f0be2a10deb4a4e81d582a054ffd317a"} Jul 17 17:58:01.583023 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.582976 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-r9p9d" event={"ID":"2cfde866-95bf-4f90-b8e1-15661fba6b87","Type":"ContainerStarted","Data":"386f1fcfd178b930ae0f3b7ef5f9fac5f498ccb409c94438753814afdf0489cc"} Jul 17 17:58:01.583201 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.583032 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-575c877f5f-4jvwf" Jul 17 17:58:01.583201 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.583069 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-r9p9d" podUID="2cfde866-95bf-4f90-b8e1-15661fba6b87" containerName="authorino" containerID="cri-o://386f1fcfd178b930ae0f3b7ef5f9fac5f498ccb409c94438753814afdf0489cc" gracePeriod=30 Jul 17 17:58:01.583201 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.583040 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-r9p9d" event={"ID":"2cfde866-95bf-4f90-b8e1-15661fba6b87","Type":"ContainerStarted","Data":"8be6a8639194525b8b2870bab66939853776d42c3034d1ad73439e7d89b6dcbe"} Jul 17 17:58:01.590449 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.590427 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-575c877f5f-4jvwf" Jul 17 17:58:01.597320 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.597253 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-r9p9d" podStartSLOduration=1.10729363 podStartE2EDuration="1.597236893s" totalImagesPullingTime="489.943263ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:58:01.595618701 +0000 UTC m=+493.183571385" watchObservedRunningTime="2026-07-17 17:58:01.597236893 +0000 UTC m=+493.185189581" Jul 17 17:58:01.714105 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.714068 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsnsk\" (UniqueName: \"kubernetes.io/projected/c836651f-7405-496d-b414-afe9f0adc7e5-kube-api-access-xsnsk\") pod \"c836651f-7405-496d-b414-afe9f0adc7e5\" (UID: \"c836651f-7405-496d-b414-afe9f0adc7e5\") " Jul 17 17:58:01.716178 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.716156 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c836651f-7405-496d-b414-afe9f0adc7e5-kube-api-access-xsnsk" (OuterVolumeSpecName: "kube-api-access-xsnsk") pod "c836651f-7405-496d-b414-afe9f0adc7e5" (UID: "c836651f-7405-496d-b414-afe9f0adc7e5"). InnerVolumeSpecName "kube-api-access-xsnsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:58:01.815498 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.815470 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsnsk\" (UniqueName: \"kubernetes.io/projected/c836651f-7405-496d-b414-afe9f0adc7e5-kube-api-access-xsnsk\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:58:01.819803 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.819777 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-r9p9d" Jul 17 17:58:01.916311 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.916270 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffvkt\" (UniqueName: \"kubernetes.io/projected/2cfde866-95bf-4f90-b8e1-15661fba6b87-kube-api-access-ffvkt\") pod \"2cfde866-95bf-4f90-b8e1-15661fba6b87\" (UID: \"2cfde866-95bf-4f90-b8e1-15661fba6b87\") " Jul 17 17:58:01.918586 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:01.918563 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cfde866-95bf-4f90-b8e1-15661fba6b87-kube-api-access-ffvkt" (OuterVolumeSpecName: "kube-api-access-ffvkt") pod "2cfde866-95bf-4f90-b8e1-15661fba6b87" (UID: "2cfde866-95bf-4f90-b8e1-15661fba6b87"). InnerVolumeSpecName "kube-api-access-ffvkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:58:02.017716 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:02.017679 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffvkt\" (UniqueName: \"kubernetes.io/projected/2cfde866-95bf-4f90-b8e1-15661fba6b87-kube-api-access-ffvkt\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:58:02.590030 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:02.589988 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cf6cf6459-xpt2h" event={"ID":"ce07bba1-873f-42d5-ad2a-464e597edd18","Type":"ContainerStarted","Data":"44bbb236895ffbf1c7bad5ad149683d66f9c15003773ac7ee675e9c14e801e26"} Jul 17 17:58:02.591205 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:02.591183 2574 generic.go:356] "Generic (PLEG): container finished" podID="2cfde866-95bf-4f90-b8e1-15661fba6b87" containerID="386f1fcfd178b930ae0f3b7ef5f9fac5f498ccb409c94438753814afdf0489cc" exitCode=2 Jul 17 17:58:02.591301 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:02.591220 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-r9p9d" Jul 17 17:58:02.591301 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:02.591233 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-575c877f5f-4jvwf" Jul 17 17:58:02.591301 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:02.591261 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-r9p9d" event={"ID":"2cfde866-95bf-4f90-b8e1-15661fba6b87","Type":"ContainerDied","Data":"386f1fcfd178b930ae0f3b7ef5f9fac5f498ccb409c94438753814afdf0489cc"} Jul 17 17:58:02.591301 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:02.591292 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-r9p9d" event={"ID":"2cfde866-95bf-4f90-b8e1-15661fba6b87","Type":"ContainerDied","Data":"8be6a8639194525b8b2870bab66939853776d42c3034d1ad73439e7d89b6dcbe"} Jul 17 17:58:02.591489 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:02.591308 2574 scope.go:117] "RemoveContainer" containerID="386f1fcfd178b930ae0f3b7ef5f9fac5f498ccb409c94438753814afdf0489cc" Jul 17 17:58:02.603693 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:02.603671 2574 scope.go:117] "RemoveContainer" containerID="386f1fcfd178b930ae0f3b7ef5f9fac5f498ccb409c94438753814afdf0489cc" Jul 17 17:58:02.603991 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:58:02.603969 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386f1fcfd178b930ae0f3b7ef5f9fac5f498ccb409c94438753814afdf0489cc\": container with ID starting with 386f1fcfd178b930ae0f3b7ef5f9fac5f498ccb409c94438753814afdf0489cc not found: ID does not exist" containerID="386f1fcfd178b930ae0f3b7ef5f9fac5f498ccb409c94438753814afdf0489cc" Jul 17 17:58:02.604078 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:02.604000 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386f1fcfd178b930ae0f3b7ef5f9fac5f498ccb409c94438753814afdf0489cc"} err="failed to get container status \"386f1fcfd178b930ae0f3b7ef5f9fac5f498ccb409c94438753814afdf0489cc\": rpc error: code = NotFound desc = could not find container \"386f1fcfd178b930ae0f3b7ef5f9fac5f498ccb409c94438753814afdf0489cc\": container with ID starting with 386f1fcfd178b930ae0f3b7ef5f9fac5f498ccb409c94438753814afdf0489cc not found: ID does not exist" Jul 17 17:58:02.605750 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:02.605706 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6cf6cf6459-xpt2h" podStartSLOduration=2.08107971 podStartE2EDuration="2.605690383s" totalImagesPullingTime="524.610673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:58:02.60438537 +0000 UTC m=+494.192338059" watchObservedRunningTime="2026-07-17 17:58:02.605690383 +0000 UTC m=+494.193643072" Jul 17 17:58:02.627165 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:02.627128 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-575c877f5f-4jvwf"] Jul 17 17:58:02.632821 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:02.632792 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-575c877f5f-4jvwf"] Jul 17 17:58:02.634653 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:02.634625 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-rlqv9"] Jul 17 17:58:02.634821 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:02.634800 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-rlqv9" podUID="87941017-722f-4f9b-9f2b-33718a2ee9ae" containerName="authorino" containerID="cri-o://dd9d290ad5d4394dee3e832de079b152f69ee437fa327aa44c48882be3594e2d" gracePeriod=30 Jul 17 17:58:02.644287 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:02.644262 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-r9p9d"] Jul 17 17:58:02.648012 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:02.647989 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-r9p9d"] Jul 17 17:58:02.888989 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:02.888968 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-rlqv9" Jul 17 17:58:02.985763 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:02.985730 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cfde866-95bf-4f90-b8e1-15661fba6b87" path="/var/lib/kubelet/pods/2cfde866-95bf-4f90-b8e1-15661fba6b87/volumes" Jul 17 17:58:02.986072 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:02.986060 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c836651f-7405-496d-b414-afe9f0adc7e5" path="/var/lib/kubelet/pods/c836651f-7405-496d-b414-afe9f0adc7e5/volumes" Jul 17 17:58:03.027074 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:03.027047 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pt2gt\" (UniqueName: \"kubernetes.io/projected/87941017-722f-4f9b-9f2b-33718a2ee9ae-kube-api-access-pt2gt\") pod \"87941017-722f-4f9b-9f2b-33718a2ee9ae\" (UID: \"87941017-722f-4f9b-9f2b-33718a2ee9ae\") " Jul 17 17:58:03.029334 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:03.029295 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87941017-722f-4f9b-9f2b-33718a2ee9ae-kube-api-access-pt2gt" (OuterVolumeSpecName: "kube-api-access-pt2gt") pod "87941017-722f-4f9b-9f2b-33718a2ee9ae" (UID: "87941017-722f-4f9b-9f2b-33718a2ee9ae"). InnerVolumeSpecName "kube-api-access-pt2gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 17:58:03.128564 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:03.128474 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt2gt\" (UniqueName: \"kubernetes.io/projected/87941017-722f-4f9b-9f2b-33718a2ee9ae-kube-api-access-pt2gt\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 17:58:03.597791 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:03.597754 2574 generic.go:356] "Generic (PLEG): container finished" podID="87941017-722f-4f9b-9f2b-33718a2ee9ae" containerID="dd9d290ad5d4394dee3e832de079b152f69ee437fa327aa44c48882be3594e2d" exitCode=0 Jul 17 17:58:03.598269 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:03.597801 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-rlqv9" Jul 17 17:58:03.598269 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:03.597840 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-rlqv9" event={"ID":"87941017-722f-4f9b-9f2b-33718a2ee9ae","Type":"ContainerDied","Data":"dd9d290ad5d4394dee3e832de079b152f69ee437fa327aa44c48882be3594e2d"} Jul 17 17:58:03.598269 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:03.597883 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-rlqv9" event={"ID":"87941017-722f-4f9b-9f2b-33718a2ee9ae","Type":"ContainerDied","Data":"1172a91bdfa34d388deb8df65144bdc979040ffeb5e08b58c19aa519a01fe63c"} Jul 17 17:58:03.598269 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:03.597900 2574 scope.go:117] "RemoveContainer" containerID="dd9d290ad5d4394dee3e832de079b152f69ee437fa327aa44c48882be3594e2d" Jul 17 17:58:03.607298 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:03.607278 2574 scope.go:117] "RemoveContainer" containerID="dd9d290ad5d4394dee3e832de079b152f69ee437fa327aa44c48882be3594e2d" Jul 17 17:58:03.607619 ip-10-0-142-155 kubenswrapper[2574]: E0717 17:58:03.607599 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd9d290ad5d4394dee3e832de079b152f69ee437fa327aa44c48882be3594e2d\": container with ID starting with dd9d290ad5d4394dee3e832de079b152f69ee437fa327aa44c48882be3594e2d not found: ID does not exist" containerID="dd9d290ad5d4394dee3e832de079b152f69ee437fa327aa44c48882be3594e2d" Jul 17 17:58:03.607695 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:03.607628 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd9d290ad5d4394dee3e832de079b152f69ee437fa327aa44c48882be3594e2d"} err="failed to get container status \"dd9d290ad5d4394dee3e832de079b152f69ee437fa327aa44c48882be3594e2d\": rpc error: code = NotFound desc = could not find container \"dd9d290ad5d4394dee3e832de079b152f69ee437fa327aa44c48882be3594e2d\": container with ID starting with dd9d290ad5d4394dee3e832de079b152f69ee437fa327aa44c48882be3594e2d not found: ID does not exist" Jul 17 17:58:03.622000 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:03.621974 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-rlqv9"] Jul 17 17:58:03.627988 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:03.627952 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-rlqv9"] Jul 17 17:58:04.985846 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:04.985803 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87941017-722f-4f9b-9f2b-33718a2ee9ae" path="/var/lib/kubelet/pods/87941017-722f-4f9b-9f2b-33718a2ee9ae/volumes" Jul 17 17:58:07.829931 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:07.829892 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:58:07.892228 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:07.892191 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:58:07.945272 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:07.945238 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:58:11.219951 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:11.219919 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:58:11.316741 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:11.316708 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:58:30.914342 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:30.914302 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:58:30.964640 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:30.964605 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:58:43.898989 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:43.898941 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:58:43.992708 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:43.992632 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:58:44.475003 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:44.474962 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:58:44.654818 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:44.654758 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:58:44.906856 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:44.906761 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:58:45.011120 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:45.011082 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:58:45.015551 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:45.015527 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-xbqzn"] Jul 17 17:58:45.015938 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:45.015925 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87941017-722f-4f9b-9f2b-33718a2ee9ae" containerName="authorino" Jul 17 17:58:45.015994 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:45.015942 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="87941017-722f-4f9b-9f2b-33718a2ee9ae" containerName="authorino" Jul 17 17:58:45.015994 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:45.015965 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cfde866-95bf-4f90-b8e1-15661fba6b87" containerName="authorino" Jul 17 17:58:45.015994 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:45.015972 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cfde866-95bf-4f90-b8e1-15661fba6b87" containerName="authorino" Jul 17 17:58:45.016118 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:45.016037 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="87941017-722f-4f9b-9f2b-33718a2ee9ae" containerName="authorino" Jul 17 17:58:45.016118 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:45.016046 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cfde866-95bf-4f90-b8e1-15661fba6b87" containerName="authorino" Jul 17 17:58:45.019425 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:45.019403 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-xbqzn" Jul 17 17:58:45.022083 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:45.022059 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 17 17:58:45.022205 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:45.022082 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-ggssl\"" Jul 17 17:58:45.028537 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:45.028513 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-xbqzn"] Jul 17 17:58:45.109670 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:45.109630 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pg4b\" (UniqueName: \"kubernetes.io/projected/c483e1a0-1ed1-4f87-a9c9-aa84e01a72f2-kube-api-access-7pg4b\") pod \"payload-pre-processing-7677b774bb-xbqzn\" (UID: \"c483e1a0-1ed1-4f87-a9c9-aa84e01a72f2\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xbqzn" Jul 17 17:58:45.109670 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:45.109673 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c483e1a0-1ed1-4f87-a9c9-aa84e01a72f2-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-xbqzn\" (UID: \"c483e1a0-1ed1-4f87-a9c9-aa84e01a72f2\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xbqzn" Jul 17 17:58:45.211708 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:45.211618 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pg4b\" (UniqueName: \"kubernetes.io/projected/c483e1a0-1ed1-4f87-a9c9-aa84e01a72f2-kube-api-access-7pg4b\") pod \"payload-pre-processing-7677b774bb-xbqzn\" (UID: \"c483e1a0-1ed1-4f87-a9c9-aa84e01a72f2\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xbqzn" Jul 17 17:58:45.211708 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:45.211669 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c483e1a0-1ed1-4f87-a9c9-aa84e01a72f2-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-xbqzn\" (UID: \"c483e1a0-1ed1-4f87-a9c9-aa84e01a72f2\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xbqzn" Jul 17 17:58:45.212338 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:45.212320 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c483e1a0-1ed1-4f87-a9c9-aa84e01a72f2-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-xbqzn\" (UID: \"c483e1a0-1ed1-4f87-a9c9-aa84e01a72f2\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xbqzn" Jul 17 17:58:45.219710 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:45.219685 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pg4b\" (UniqueName: \"kubernetes.io/projected/c483e1a0-1ed1-4f87-a9c9-aa84e01a72f2-kube-api-access-7pg4b\") pod \"payload-pre-processing-7677b774bb-xbqzn\" (UID: \"c483e1a0-1ed1-4f87-a9c9-aa84e01a72f2\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-xbqzn" Jul 17 17:58:45.332377 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:45.332310 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-xbqzn" Jul 17 17:58:45.492664 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:45.492639 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-xbqzn"] Jul 17 17:58:45.494469 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:58:45.494441 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc483e1a0_1ed1_4f87_a9c9_aa84e01a72f2.slice/crio-ebd24fc59c6e04c9d8d943242862307aaab6d252f98122aef21df9984a626201 WatchSource:0}: Error finding container ebd24fc59c6e04c9d8d943242862307aaab6d252f98122aef21df9984a626201: Status 404 returned error can't find the container with id ebd24fc59c6e04c9d8d943242862307aaab6d252f98122aef21df9984a626201 Jul 17 17:58:45.803025 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:45.802930 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-xbqzn" event={"ID":"c483e1a0-1ed1-4f87-a9c9-aa84e01a72f2","Type":"ContainerStarted","Data":"ebd24fc59c6e04c9d8d943242862307aaab6d252f98122aef21df9984a626201"} Jul 17 17:58:47.816393 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:47.816342 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-xbqzn" event={"ID":"c483e1a0-1ed1-4f87-a9c9-aa84e01a72f2","Type":"ContainerStarted","Data":"734cc580383d2c2a874d0e7438309c8f014036af9795c55df2c36485a2c388d4"} Jul 17 17:58:47.816832 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:47.816420 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-xbqzn" Jul 17 17:58:47.833590 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:47.833534 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-xbqzn" podStartSLOduration=2.293685324 podStartE2EDuration="3.833516833s" totalImagesPullingTime="1.539831509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:58:47.833033248 +0000 UTC m=+539.420985955" watchObservedRunningTime="2026-07-17 17:58:47.833516833 +0000 UTC m=+539.421469580" Jul 17 17:58:49.302943 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:49.302911 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:58:49.355226 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:49.355193 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:58:54.376817 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:54.376778 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:58:54.474160 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:54.474126 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:58:58.823539 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:58:58.823510 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-xbqzn" Jul 17 17:59:04.223840 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.223803 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m"] Jul 17 17:59:04.228915 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.228894 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:04.232773 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.232738 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fv28k\"" Jul 17 17:59:04.232921 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.232782 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 17 17:59:04.232921 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.232846 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 17 17:59:04.232921 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.232782 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 17 17:59:04.237140 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.236690 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m"] Jul 17 17:59:04.253196 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.253161 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:04.402633 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.402557 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/875c5d7b-aa74-459a-a3fe-8b9bac191aea-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m\" (UID: \"875c5d7b-aa74-459a-a3fe-8b9bac191aea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:04.402817 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.402724 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/875c5d7b-aa74-459a-a3fe-8b9bac191aea-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m\" (UID: \"875c5d7b-aa74-459a-a3fe-8b9bac191aea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:04.402817 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.402774 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/875c5d7b-aa74-459a-a3fe-8b9bac191aea-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m\" (UID: \"875c5d7b-aa74-459a-a3fe-8b9bac191aea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:04.402817 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.402808 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/875c5d7b-aa74-459a-a3fe-8b9bac191aea-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m\" (UID: \"875c5d7b-aa74-459a-a3fe-8b9bac191aea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:04.402997 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.402835 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/875c5d7b-aa74-459a-a3fe-8b9bac191aea-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m\" (UID: \"875c5d7b-aa74-459a-a3fe-8b9bac191aea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:04.402997 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.402886 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlstn\" (UniqueName: \"kubernetes.io/projected/875c5d7b-aa74-459a-a3fe-8b9bac191aea-kube-api-access-dlstn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m\" (UID: \"875c5d7b-aa74-459a-a3fe-8b9bac191aea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:04.504276 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.504178 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/875c5d7b-aa74-459a-a3fe-8b9bac191aea-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m\" (UID: \"875c5d7b-aa74-459a-a3fe-8b9bac191aea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:04.504276 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.504239 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/875c5d7b-aa74-459a-a3fe-8b9bac191aea-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m\" (UID: \"875c5d7b-aa74-459a-a3fe-8b9bac191aea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:04.504276 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.504263 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/875c5d7b-aa74-459a-a3fe-8b9bac191aea-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m\" (UID: \"875c5d7b-aa74-459a-a3fe-8b9bac191aea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:04.504595 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.504289 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlstn\" (UniqueName: \"kubernetes.io/projected/875c5d7b-aa74-459a-a3fe-8b9bac191aea-kube-api-access-dlstn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m\" (UID: \"875c5d7b-aa74-459a-a3fe-8b9bac191aea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:04.504595 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.504341 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/875c5d7b-aa74-459a-a3fe-8b9bac191aea-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m\" (UID: \"875c5d7b-aa74-459a-a3fe-8b9bac191aea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:04.504595 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.504454 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/875c5d7b-aa74-459a-a3fe-8b9bac191aea-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m\" (UID: \"875c5d7b-aa74-459a-a3fe-8b9bac191aea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:04.504778 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.504749 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/875c5d7b-aa74-459a-a3fe-8b9bac191aea-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m\" (UID: \"875c5d7b-aa74-459a-a3fe-8b9bac191aea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:04.504778 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.504762 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/875c5d7b-aa74-459a-a3fe-8b9bac191aea-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m\" (UID: \"875c5d7b-aa74-459a-a3fe-8b9bac191aea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:04.504891 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.504837 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/875c5d7b-aa74-459a-a3fe-8b9bac191aea-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m\" (UID: \"875c5d7b-aa74-459a-a3fe-8b9bac191aea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:04.506672 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.506650 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/875c5d7b-aa74-459a-a3fe-8b9bac191aea-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m\" (UID: \"875c5d7b-aa74-459a-a3fe-8b9bac191aea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:04.506928 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.506908 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/875c5d7b-aa74-459a-a3fe-8b9bac191aea-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m\" (UID: \"875c5d7b-aa74-459a-a3fe-8b9bac191aea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:04.513294 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.513254 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlstn\" (UniqueName: \"kubernetes.io/projected/875c5d7b-aa74-459a-a3fe-8b9bac191aea-kube-api-access-dlstn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m\" (UID: \"875c5d7b-aa74-459a-a3fe-8b9bac191aea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:04.541108 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.541074 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:04.685554 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.685530 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m"] Jul 17 17:59:04.688143 ip-10-0-142-155 kubenswrapper[2574]: W0717 17:59:04.688113 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875c5d7b_aa74_459a_a3fe_8b9bac191aea.slice/crio-b376818ab74d580d9550836093e0a9303b4336696a6d7e69d685151f246563cc WatchSource:0}: Error finding container b376818ab74d580d9550836093e0a9303b4336696a6d7e69d685151f246563cc: Status 404 returned error can't find the container with id b376818ab74d580d9550836093e0a9303b4336696a6d7e69d685151f246563cc Jul 17 17:59:04.902919 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:04.902825 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" event={"ID":"875c5d7b-aa74-459a-a3fe-8b9bac191aea","Type":"ContainerStarted","Data":"b376818ab74d580d9550836093e0a9303b4336696a6d7e69d685151f246563cc"} Jul 17 17:59:05.910860 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:05.910825 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" event={"ID":"875c5d7b-aa74-459a-a3fe-8b9bac191aea","Type":"ContainerStarted","Data":"c918c0a252b27ccc19760c5be9c6173b841755a6116090ba2c4bb2b9d29e5743"} Jul 17 17:59:05.927130 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:05.926959 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" podStartSLOduration=0.807451966 podStartE2EDuration="1.926936624s" totalImagesPullingTime="1.119484658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 17:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 17:59:05.926530312 +0000 UTC m=+557.514483031" watchObservedRunningTime="2026-07-17 17:59:05.926936624 +0000 UTC m=+557.514889316" Jul 17 17:59:10.494766 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:10.494737 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:14.541610 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:14.541571 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:14.542049 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:14.541637 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:14.554623 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:14.554594 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:14.966251 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:14.966221 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m" Jul 17 17:59:17.629210 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:17.629180 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:23.137539 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:23.137509 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:34.333863 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:34.333831 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:40.328279 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:40.328247 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 17:59:40.933313 ip-10-0-142-155 kubenswrapper[2574]: I0717 17:59:40.933271 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:00:00.140822 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:00.140787 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738520-929bz"] Jul 17 18:00:00.144547 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:00.144529 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738520-929bz" Jul 17 18:00:00.146861 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:00.146841 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dpsm\"" Jul 17 18:00:00.151076 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:00.151013 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738520-929bz"] Jul 17 18:00:00.156127 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:00.156106 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738520-929bz" Jul 17 18:00:00.293548 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:00.293515 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738520-929bz"] Jul 17 18:00:00.296347 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:00:00.296322 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a161bd7_b2ed_4618_bfcc_528b681b9353.slice/crio-42d0d96114665dd3fd2c987c5bbbe2560770c1cf099df55fb4e66dcc4f292260 WatchSource:0}: Error finding container 42d0d96114665dd3fd2c987c5bbbe2560770c1cf099df55fb4e66dcc4f292260: Status 404 returned error can't find the container with id 42d0d96114665dd3fd2c987c5bbbe2560770c1cf099df55fb4e66dcc4f292260 Jul 17 18:00:01.190963 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:01.190929 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738520-929bz" event={"ID":"2a161bd7-b2ed-4618-bfcc-528b681b9353","Type":"ContainerStarted","Data":"42d0d96114665dd3fd2c987c5bbbe2560770c1cf099df55fb4e66dcc4f292260"} Jul 17 18:00:06.219031 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:06.218993 2574 generic.go:356] "Generic (PLEG): container finished" podID="2a161bd7-b2ed-4618-bfcc-528b681b9353" containerID="faf2a644ec36a7d551743e18b96fca7df3a29018d90965793cedf07fb4a9b71a" exitCode=0 Jul 17 18:00:06.219527 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:06.219082 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738520-929bz" event={"ID":"2a161bd7-b2ed-4618-bfcc-528b681b9353","Type":"ContainerDied","Data":"faf2a644ec36a7d551743e18b96fca7df3a29018d90965793cedf07fb4a9b71a"} Jul 17 18:00:07.359131 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:07.359099 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738520-929bz" Jul 17 18:00:08.229653 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:08.229622 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738520-929bz" Jul 17 18:00:08.229845 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:08.229661 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738520-929bz" event={"ID":"2a161bd7-b2ed-4618-bfcc-528b681b9353","Type":"ContainerDied","Data":"42d0d96114665dd3fd2c987c5bbbe2560770c1cf099df55fb4e66dcc4f292260"} Jul 17 18:00:08.229845 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:08.229706 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42d0d96114665dd3fd2c987c5bbbe2560770c1cf099df55fb4e66dcc4f292260" Jul 17 18:00:31.987261 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:31.987223 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8f9c7c886-rc5n9"] Jul 17 18:00:31.987692 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:31.987661 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a161bd7-b2ed-4618-bfcc-528b681b9353" containerName="cleanup" Jul 17 18:00:31.987692 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:31.987674 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a161bd7-b2ed-4618-bfcc-528b681b9353" containerName="cleanup" Jul 17 18:00:31.987779 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:31.987745 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a161bd7-b2ed-4618-bfcc-528b681b9353" containerName="cleanup" Jul 17 18:00:31.990990 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:31.990968 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8f9c7c886-rc5n9" Jul 17 18:00:31.999948 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:31.999923 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8f9c7c886-rc5n9"] Jul 17 18:00:32.044411 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:32.044380 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/17141199-6d8a-4bcd-be1b-c00b1b7c6b29-tls-cert\") pod \"authorino-8f9c7c886-rc5n9\" (UID: \"17141199-6d8a-4bcd-be1b-c00b1b7c6b29\") " pod="kuadrant-system/authorino-8f9c7c886-rc5n9" Jul 17 18:00:32.044600 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:32.044487 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxrwf\" (UniqueName: \"kubernetes.io/projected/17141199-6d8a-4bcd-be1b-c00b1b7c6b29-kube-api-access-hxrwf\") pod \"authorino-8f9c7c886-rc5n9\" (UID: \"17141199-6d8a-4bcd-be1b-c00b1b7c6b29\") " pod="kuadrant-system/authorino-8f9c7c886-rc5n9" Jul 17 18:00:32.145762 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:32.145729 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxrwf\" (UniqueName: \"kubernetes.io/projected/17141199-6d8a-4bcd-be1b-c00b1b7c6b29-kube-api-access-hxrwf\") pod \"authorino-8f9c7c886-rc5n9\" (UID: \"17141199-6d8a-4bcd-be1b-c00b1b7c6b29\") " pod="kuadrant-system/authorino-8f9c7c886-rc5n9" Jul 17 18:00:32.145981 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:32.145831 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/17141199-6d8a-4bcd-be1b-c00b1b7c6b29-tls-cert\") pod \"authorino-8f9c7c886-rc5n9\" (UID: \"17141199-6d8a-4bcd-be1b-c00b1b7c6b29\") " pod="kuadrant-system/authorino-8f9c7c886-rc5n9" Jul 17 18:00:32.148438 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:32.148406 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/17141199-6d8a-4bcd-be1b-c00b1b7c6b29-tls-cert\") pod \"authorino-8f9c7c886-rc5n9\" (UID: \"17141199-6d8a-4bcd-be1b-c00b1b7c6b29\") " pod="kuadrant-system/authorino-8f9c7c886-rc5n9" Jul 17 18:00:32.153310 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:32.153289 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxrwf\" (UniqueName: \"kubernetes.io/projected/17141199-6d8a-4bcd-be1b-c00b1b7c6b29-kube-api-access-hxrwf\") pod \"authorino-8f9c7c886-rc5n9\" (UID: \"17141199-6d8a-4bcd-be1b-c00b1b7c6b29\") " pod="kuadrant-system/authorino-8f9c7c886-rc5n9" Jul 17 18:00:32.301596 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:32.301503 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8f9c7c886-rc5n9" Jul 17 18:00:32.416034 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:32.415954 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:00:32.444668 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:32.444581 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8f9c7c886-rc5n9"] Jul 17 18:00:32.447217 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:00:32.447183 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17141199_6d8a_4bcd_be1b_c00b1b7c6b29.slice/crio-045e6eae12e2ccc3bf82a7ba7a300116c48d17bde12ac2e8ca5f0edd7ab1623b WatchSource:0}: Error finding container 045e6eae12e2ccc3bf82a7ba7a300116c48d17bde12ac2e8ca5f0edd7ab1623b: Status 404 returned error can't find the container with id 045e6eae12e2ccc3bf82a7ba7a300116c48d17bde12ac2e8ca5f0edd7ab1623b Jul 17 18:00:32.511061 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:32.511029 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:00:33.355808 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:33.355764 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8f9c7c886-rc5n9" event={"ID":"17141199-6d8a-4bcd-be1b-c00b1b7c6b29","Type":"ContainerStarted","Data":"60ed0db76b3b3e646421dac2f6dd523047b298179c3a8866724998bcd95b643e"} Jul 17 18:00:33.356205 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:33.355818 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8f9c7c886-rc5n9" event={"ID":"17141199-6d8a-4bcd-be1b-c00b1b7c6b29","Type":"ContainerStarted","Data":"045e6eae12e2ccc3bf82a7ba7a300116c48d17bde12ac2e8ca5f0edd7ab1623b"} Jul 17 18:00:33.378101 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:33.378027 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8f9c7c886-rc5n9" podStartSLOduration=1.94080919 podStartE2EDuration="2.378002963s" totalImagesPullingTime="437.193773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:00:33.375422327 +0000 UTC m=+644.963375023" watchObservedRunningTime="2026-07-17 18:00:33.378002963 +0000 UTC m=+644.965955653" Jul 17 18:00:33.405635 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:33.405593 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cf6cf6459-xpt2h"] Jul 17 18:00:33.405833 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:33.405776 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6cf6cf6459-xpt2h" podUID="ce07bba1-873f-42d5-ad2a-464e597edd18" containerName="authorino" containerID="cri-o://44bbb236895ffbf1c7bad5ad149683d66f9c15003773ac7ee675e9c14e801e26" gracePeriod=30 Jul 17 18:00:33.625391 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:33.625256 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:00:33.670368 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:33.670321 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cf6cf6459-xpt2h" Jul 17 18:00:33.720734 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:33.720697 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:00:33.758028 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:33.757988 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts79k\" (UniqueName: \"kubernetes.io/projected/ce07bba1-873f-42d5-ad2a-464e597edd18-kube-api-access-ts79k\") pod \"ce07bba1-873f-42d5-ad2a-464e597edd18\" (UID: \"ce07bba1-873f-42d5-ad2a-464e597edd18\") " Jul 17 18:00:33.758236 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:33.758090 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ce07bba1-873f-42d5-ad2a-464e597edd18-tls-cert\") pod \"ce07bba1-873f-42d5-ad2a-464e597edd18\" (UID: \"ce07bba1-873f-42d5-ad2a-464e597edd18\") " Jul 17 18:00:33.760509 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:33.760480 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce07bba1-873f-42d5-ad2a-464e597edd18-kube-api-access-ts79k" (OuterVolumeSpecName: "kube-api-access-ts79k") pod "ce07bba1-873f-42d5-ad2a-464e597edd18" (UID: "ce07bba1-873f-42d5-ad2a-464e597edd18"). InnerVolumeSpecName "kube-api-access-ts79k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:00:33.770951 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:33.770923 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce07bba1-873f-42d5-ad2a-464e597edd18-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ce07bba1-873f-42d5-ad2a-464e597edd18" (UID: "ce07bba1-873f-42d5-ad2a-464e597edd18"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:00:33.858929 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:33.858885 2574 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ce07bba1-873f-42d5-ad2a-464e597edd18-tls-cert\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:00:33.858929 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:33.858929 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts79k\" (UniqueName: \"kubernetes.io/projected/ce07bba1-873f-42d5-ad2a-464e597edd18-kube-api-access-ts79k\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:00:34.362482 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:34.362444 2574 generic.go:356] "Generic (PLEG): container finished" podID="ce07bba1-873f-42d5-ad2a-464e597edd18" containerID="44bbb236895ffbf1c7bad5ad149683d66f9c15003773ac7ee675e9c14e801e26" exitCode=0 Jul 17 18:00:34.362932 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:34.362504 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cf6cf6459-xpt2h" Jul 17 18:00:34.362932 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:34.362524 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cf6cf6459-xpt2h" event={"ID":"ce07bba1-873f-42d5-ad2a-464e597edd18","Type":"ContainerDied","Data":"44bbb236895ffbf1c7bad5ad149683d66f9c15003773ac7ee675e9c14e801e26"} Jul 17 18:00:34.362932 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:34.362563 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cf6cf6459-xpt2h" event={"ID":"ce07bba1-873f-42d5-ad2a-464e597edd18","Type":"ContainerDied","Data":"4f6a83614c213847958671023e2c4ef7f0be2a10deb4a4e81d582a054ffd317a"} Jul 17 18:00:34.362932 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:34.362579 2574 scope.go:117] "RemoveContainer" containerID="44bbb236895ffbf1c7bad5ad149683d66f9c15003773ac7ee675e9c14e801e26" Jul 17 18:00:34.372311 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:34.372293 2574 scope.go:117] "RemoveContainer" containerID="44bbb236895ffbf1c7bad5ad149683d66f9c15003773ac7ee675e9c14e801e26" Jul 17 18:00:34.372650 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:00:34.372626 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44bbb236895ffbf1c7bad5ad149683d66f9c15003773ac7ee675e9c14e801e26\": container with ID starting with 44bbb236895ffbf1c7bad5ad149683d66f9c15003773ac7ee675e9c14e801e26 not found: ID does not exist" containerID="44bbb236895ffbf1c7bad5ad149683d66f9c15003773ac7ee675e9c14e801e26" Jul 17 18:00:34.372783 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:34.372657 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44bbb236895ffbf1c7bad5ad149683d66f9c15003773ac7ee675e9c14e801e26"} err="failed to get container status \"44bbb236895ffbf1c7bad5ad149683d66f9c15003773ac7ee675e9c14e801e26\": rpc error: code = NotFound desc = could not find container \"44bbb236895ffbf1c7bad5ad149683d66f9c15003773ac7ee675e9c14e801e26\": container with ID starting with 44bbb236895ffbf1c7bad5ad149683d66f9c15003773ac7ee675e9c14e801e26 not found: ID does not exist" Jul 17 18:00:34.387870 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:34.387838 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cf6cf6459-xpt2h"] Jul 17 18:00:34.393683 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:34.393656 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6cf6cf6459-xpt2h"] Jul 17 18:00:34.986254 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:00:34.986224 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce07bba1-873f-42d5-ad2a-464e597edd18" path="/var/lib/kubelet/pods/ce07bba1-873f-42d5-ad2a-464e597edd18/volumes" Jul 17 18:01:38.061394 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:01:38.060214 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:37.979222 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:02:37.979190 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:39.262872 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:02:39.262838 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:02:40.838093 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:02:40.838062 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:03:05.657685 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:03:05.657642 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:03:05.758040 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:03:05.758007 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:03:07.817465 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:03:07.817429 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:03:07.944888 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:03:07.944851 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:03:08.098744 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:03:08.098653 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:03:13.769309 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:03:13.769225 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:03:14.018603 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:03:14.018569 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:03:14.094043 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:03:14.093948 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:03:14.192999 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:03:14.192972 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:03:19.958753 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:03:19.958722 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:03:20.018200 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:03:20.018162 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:04:41.285499 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:04:41.285470 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:50.042694 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:06:50.042653 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:06:50.127904 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:06:50.127872 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:07:19.576344 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:07:19.576306 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td"] Jul 17 18:07:19.576790 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:07:19.576523 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td" podUID="1cd3e533-bd47-43dc-b45d-32f3ff4827df" containerName="manager" containerID="cri-o://264c3f8efa5e44af5e3264b0062b5f16cf265e1287984cb88939838c31c72e99" gracePeriod=10 Jul 17 18:07:21.938945 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:07:21.938916 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td" Jul 17 18:07:22.040597 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:07:22.040501 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bbfd\" (UniqueName: \"kubernetes.io/projected/1cd3e533-bd47-43dc-b45d-32f3ff4827df-kube-api-access-8bbfd\") pod \"1cd3e533-bd47-43dc-b45d-32f3ff4827df\" (UID: \"1cd3e533-bd47-43dc-b45d-32f3ff4827df\") " Jul 17 18:07:22.040597 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:07:22.040562 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1cd3e533-bd47-43dc-b45d-32f3ff4827df-extensions-socket-volume\") pod \"1cd3e533-bd47-43dc-b45d-32f3ff4827df\" (UID: \"1cd3e533-bd47-43dc-b45d-32f3ff4827df\") " Jul 17 18:07:22.040938 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:07:22.040913 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd3e533-bd47-43dc-b45d-32f3ff4827df-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1cd3e533-bd47-43dc-b45d-32f3ff4827df" (UID: "1cd3e533-bd47-43dc-b45d-32f3ff4827df"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:07:22.042858 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:07:22.042834 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd3e533-bd47-43dc-b45d-32f3ff4827df-kube-api-access-8bbfd" (OuterVolumeSpecName: "kube-api-access-8bbfd") pod "1cd3e533-bd47-43dc-b45d-32f3ff4827df" (UID: "1cd3e533-bd47-43dc-b45d-32f3ff4827df"). InnerVolumeSpecName "kube-api-access-8bbfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:07:22.142318 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:07:22.142276 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bbfd\" (UniqueName: \"kubernetes.io/projected/1cd3e533-bd47-43dc-b45d-32f3ff4827df-kube-api-access-8bbfd\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:07:22.142318 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:07:22.142309 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1cd3e533-bd47-43dc-b45d-32f3ff4827df-extensions-socket-volume\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:07:22.428325 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:07:22.428233 2574 generic.go:356] "Generic (PLEG): container finished" podID="1cd3e533-bd47-43dc-b45d-32f3ff4827df" containerID="264c3f8efa5e44af5e3264b0062b5f16cf265e1287984cb88939838c31c72e99" exitCode=0 Jul 17 18:07:22.428325 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:07:22.428277 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td" Jul 17 18:07:22.428325 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:07:22.428308 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td" event={"ID":"1cd3e533-bd47-43dc-b45d-32f3ff4827df","Type":"ContainerDied","Data":"264c3f8efa5e44af5e3264b0062b5f16cf265e1287984cb88939838c31c72e99"} Jul 17 18:07:22.428570 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:07:22.428373 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td" event={"ID":"1cd3e533-bd47-43dc-b45d-32f3ff4827df","Type":"ContainerDied","Data":"8ad34a1e695b138778b5d22ec204a3e809719050d8367c8fac0a7152316bed3b"} Jul 17 18:07:22.428570 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:07:22.428397 2574 scope.go:117] "RemoveContainer" containerID="264c3f8efa5e44af5e3264b0062b5f16cf265e1287984cb88939838c31c72e99" Jul 17 18:07:22.438114 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:07:22.438096 2574 scope.go:117] "RemoveContainer" containerID="264c3f8efa5e44af5e3264b0062b5f16cf265e1287984cb88939838c31c72e99" Jul 17 18:07:22.438407 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:07:22.438387 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"264c3f8efa5e44af5e3264b0062b5f16cf265e1287984cb88939838c31c72e99\": container with ID starting with 264c3f8efa5e44af5e3264b0062b5f16cf265e1287984cb88939838c31c72e99 not found: ID does not exist" containerID="264c3f8efa5e44af5e3264b0062b5f16cf265e1287984cb88939838c31c72e99" Jul 17 18:07:22.438528 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:07:22.438420 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"264c3f8efa5e44af5e3264b0062b5f16cf265e1287984cb88939838c31c72e99"} err="failed to get container status \"264c3f8efa5e44af5e3264b0062b5f16cf265e1287984cb88939838c31c72e99\": rpc error: code = NotFound desc = could not find container \"264c3f8efa5e44af5e3264b0062b5f16cf265e1287984cb88939838c31c72e99\": container with ID starting with 264c3f8efa5e44af5e3264b0062b5f16cf265e1287984cb88939838c31c72e99 not found: ID does not exist" Jul 17 18:07:22.450406 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:07:22.450372 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td"] Jul 17 18:07:22.456099 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:07:22.456070 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pl6td"] Jul 17 18:07:22.985910 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:07:22.985875 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cd3e533-bd47-43dc-b45d-32f3ff4827df" path="/var/lib/kubelet/pods/1cd3e533-bd47-43dc-b45d-32f3ff4827df/volumes" Jul 17 18:08:20.761843 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:20.761804 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s242v"] Jul 17 18:08:20.762416 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:20.762205 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cd3e533-bd47-43dc-b45d-32f3ff4827df" containerName="manager" Jul 17 18:08:20.762416 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:20.762215 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cd3e533-bd47-43dc-b45d-32f3ff4827df" containerName="manager" Jul 17 18:08:20.762416 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:20.762230 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce07bba1-873f-42d5-ad2a-464e597edd18" containerName="authorino" Jul 17 18:08:20.762416 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:20.762235 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce07bba1-873f-42d5-ad2a-464e597edd18" containerName="authorino" Jul 17 18:08:20.762416 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:20.762292 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cd3e533-bd47-43dc-b45d-32f3ff4827df" containerName="manager" Jul 17 18:08:20.762416 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:20.762299 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce07bba1-873f-42d5-ad2a-464e597edd18" containerName="authorino" Jul 17 18:08:20.765435 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:20.765417 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s242v" Jul 17 18:08:20.768889 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:20.768860 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hc8h4\"" Jul 17 18:08:20.784831 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:20.784788 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/00dc5a78-7403-4a3f-850e-1ec3f63a6e6d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-s242v\" (UID: \"00dc5a78-7403-4a3f-850e-1ec3f63a6e6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s242v" Jul 17 18:08:20.784991 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:20.784897 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhj4p\" (UniqueName: \"kubernetes.io/projected/00dc5a78-7403-4a3f-850e-1ec3f63a6e6d-kube-api-access-vhj4p\") pod \"kuadrant-operator-controller-manager-86878c769f-s242v\" (UID: \"00dc5a78-7403-4a3f-850e-1ec3f63a6e6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s242v" Jul 17 18:08:20.784991 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:20.784955 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s242v"] Jul 17 18:08:20.885636 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:20.885589 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhj4p\" (UniqueName: \"kubernetes.io/projected/00dc5a78-7403-4a3f-850e-1ec3f63a6e6d-kube-api-access-vhj4p\") pod \"kuadrant-operator-controller-manager-86878c769f-s242v\" (UID: \"00dc5a78-7403-4a3f-850e-1ec3f63a6e6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s242v" Jul 17 18:08:20.885820 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:20.885714 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/00dc5a78-7403-4a3f-850e-1ec3f63a6e6d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-s242v\" (UID: \"00dc5a78-7403-4a3f-850e-1ec3f63a6e6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s242v" Jul 17 18:08:20.886073 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:20.886056 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/00dc5a78-7403-4a3f-850e-1ec3f63a6e6d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-s242v\" (UID: \"00dc5a78-7403-4a3f-850e-1ec3f63a6e6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s242v" Jul 17 18:08:20.898372 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:20.898331 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhj4p\" (UniqueName: \"kubernetes.io/projected/00dc5a78-7403-4a3f-850e-1ec3f63a6e6d-kube-api-access-vhj4p\") pod \"kuadrant-operator-controller-manager-86878c769f-s242v\" (UID: \"00dc5a78-7403-4a3f-850e-1ec3f63a6e6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s242v" Jul 17 18:08:21.076145 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:21.076054 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s242v" Jul 17 18:08:21.224686 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:21.224661 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s242v"] Jul 17 18:08:21.227293 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:08:21.227260 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00dc5a78_7403_4a3f_850e_1ec3f63a6e6d.slice/crio-6dcb3b20fd9061e8f253f690e7f168deaa457531593ccb5744414bae8d105946 WatchSource:0}: Error finding container 6dcb3b20fd9061e8f253f690e7f168deaa457531593ccb5744414bae8d105946: Status 404 returned error can't find the container with id 6dcb3b20fd9061e8f253f690e7f168deaa457531593ccb5744414bae8d105946 Jul 17 18:08:21.735149 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:21.735115 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s242v" event={"ID":"00dc5a78-7403-4a3f-850e-1ec3f63a6e6d","Type":"ContainerStarted","Data":"3bba33e3becbb32f3802c314c8e597e17af25a5170ece26ea568cd1505615fc5"} Jul 17 18:08:21.735398 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:21.735175 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s242v" event={"ID":"00dc5a78-7403-4a3f-850e-1ec3f63a6e6d","Type":"ContainerStarted","Data":"6dcb3b20fd9061e8f253f690e7f168deaa457531593ccb5744414bae8d105946"} Jul 17 18:08:21.735398 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:21.735246 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s242v" Jul 17 18:08:21.755624 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:21.755570 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s242v" podStartSLOduration=1.75555207 podStartE2EDuration="1.75555207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:08:21.753078624 +0000 UTC m=+1113.341031318" watchObservedRunningTime="2026-07-17 18:08:21.75555207 +0000 UTC m=+1113.343504759" Jul 17 18:08:32.742588 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:08:32.742556 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-s242v" Jul 17 18:10:14.868709 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:10:14.868674 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:15.991196 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:10:15.991161 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:10:49.713691 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:10:49.713654 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:11:30.082622 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:11:30.082584 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:14.096572 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:14.096529 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:23.038632 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:23.038591 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:23.493871 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:23.493837 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:23.593947 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:23.593908 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:31.321372 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:31.321321 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:36.462269 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:36.462226 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:39.519135 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:39.519096 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:39.614955 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:39.614923 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:39.984003 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:39.983961 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:46.136927 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:46.136892 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:46.201896 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:46.201866 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:46.344830 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:46.344795 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:46.407291 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:46.407202 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:47.599869 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:47.599839 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:47.702674 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:47.702637 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:47.798222 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:47.798187 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:48.288720 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:48.288671 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:48.388919 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:48.388887 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:48.489569 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:48.489533 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:50.173149 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:50.173113 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:14:52.534234 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:52.534201 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h"] Jul 17 18:14:52.537956 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:52.537932 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h" Jul 17 18:14:52.540595 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:52.540570 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dpsm\"" Jul 17 18:14:52.540745 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:52.540597 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:14:52.544941 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:52.544911 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h"] Jul 17 18:14:52.611202 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:52.611160 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1173c07a-0d1f-4e3b-a832-3879cc23b935-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h\" (UID: \"1173c07a-0d1f-4e3b-a832-3879cc23b935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h" Jul 17 18:14:52.712537 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:52.712484 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1173c07a-0d1f-4e3b-a832-3879cc23b935-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h\" (UID: \"1173c07a-0d1f-4e3b-a832-3879cc23b935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h" Jul 17 18:14:52.713133 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:52.713114 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1173c07a-0d1f-4e3b-a832-3879cc23b935-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h\" (UID: \"1173c07a-0d1f-4e3b-a832-3879cc23b935\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h" Jul 17 18:14:52.852484 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:52.852388 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h" Jul 17 18:14:52.990847 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:52.990823 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h"] Jul 17 18:14:52.993847 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:14:52.993820 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1173c07a_0d1f_4e3b_a832_3879cc23b935.slice/crio-ea33c360998611a98863a7b6c1b448551e9dc4a1610622dd6632c1165a7ff3df WatchSource:0}: Error finding container ea33c360998611a98863a7b6c1b448551e9dc4a1610622dd6632c1165a7ff3df: Status 404 returned error can't find the container with id ea33c360998611a98863a7b6c1b448551e9dc4a1610622dd6632c1165a7ff3df Jul 17 18:14:53.740788 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:53.740753 2574 generic.go:356] "Generic (PLEG): container finished" podID="1173c07a-0d1f-4e3b-a832-3879cc23b935" containerID="2cdf8b0950673d364382266e1e54f72c2b4281a8120f9e0faf6bbd962bc86103" exitCode=7 Jul 17 18:14:53.741222 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:53.740839 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h" event={"ID":"1173c07a-0d1f-4e3b-a832-3879cc23b935","Type":"ContainerDied","Data":"2cdf8b0950673d364382266e1e54f72c2b4281a8120f9e0faf6bbd962bc86103"} Jul 17 18:14:53.741222 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:53.740889 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h" event={"ID":"1173c07a-0d1f-4e3b-a832-3879cc23b935","Type":"ContainerStarted","Data":"ea33c360998611a98863a7b6c1b448551e9dc4a1610622dd6632c1165a7ff3df"} Jul 17 18:14:53.741222 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:53.740989 2574 scope.go:117] "RemoveContainer" containerID="2cdf8b0950673d364382266e1e54f72c2b4281a8120f9e0faf6bbd962bc86103" Jul 17 18:14:54.749786 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:54.749740 2574 generic.go:356] "Generic (PLEG): container finished" podID="1173c07a-0d1f-4e3b-a832-3879cc23b935" containerID="e03960d8f01ee8239806ceeb5cb9b8a7fb42d3e6f3dc111ca86af3332dc94e50" exitCode=7 Jul 17 18:14:54.750217 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:54.749808 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h" event={"ID":"1173c07a-0d1f-4e3b-a832-3879cc23b935","Type":"ContainerDied","Data":"e03960d8f01ee8239806ceeb5cb9b8a7fb42d3e6f3dc111ca86af3332dc94e50"} Jul 17 18:14:54.750217 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:54.749903 2574 scope.go:117] "RemoveContainer" containerID="2cdf8b0950673d364382266e1e54f72c2b4281a8120f9e0faf6bbd962bc86103" Jul 17 18:14:54.750348 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:54.750226 2574 scope.go:117] "RemoveContainer" containerID="e03960d8f01ee8239806ceeb5cb9b8a7fb42d3e6f3dc111ca86af3332dc94e50" Jul 17 18:14:54.750535 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:14:54.750513 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h_odh-ai-gateway-infra(1173c07a-0d1f-4e3b-a832-3879cc23b935)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h" podUID="1173c07a-0d1f-4e3b-a832-3879cc23b935" Jul 17 18:14:55.756094 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:14:55.756068 2574 scope.go:117] "RemoveContainer" containerID="e03960d8f01ee8239806ceeb5cb9b8a7fb42d3e6f3dc111ca86af3332dc94e50" Jul 17 18:14:55.756538 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:14:55.756258 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h_odh-ai-gateway-infra(1173c07a-0d1f-4e3b-a832-3879cc23b935)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h" podUID="1173c07a-0d1f-4e3b-a832-3879cc23b935" Jul 17 18:15:00.129588 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:00.129555 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-284e9f2b-29738535-jbzn8"] Jul 17 18:15:00.134467 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:00.134445 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738535-d9vlp"] Jul 17 18:15:00.134593 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:00.134570 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-284e9f2b-29738535-jbzn8" Jul 17 18:15:00.137933 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:00.137906 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738535-d9vlp" Jul 17 18:15:00.144849 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:00.144808 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738535-d9vlp"] Jul 17 18:15:00.145151 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:00.145134 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-284e9f2b-29738535-jbzn8"] Jul 17 18:15:00.159887 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:00.159859 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-284e9f2b-29738535-jbzn8" Jul 17 18:15:00.167207 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:00.167184 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738535-d9vlp" Jul 17 18:15:00.338330 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:00.338298 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738535-d9vlp"] Jul 17 18:15:00.340216 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:15:00.340181 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd448bbf4_fa79_48e3_9312_b2e64d74478c.slice/crio-96d5ff7d0153c0a45835dc2c26840a8a6bb932ddbcd3ff313f523a1ae23417e8 WatchSource:0}: Error finding container 96d5ff7d0153c0a45835dc2c26840a8a6bb932ddbcd3ff313f523a1ae23417e8: Status 404 returned error can't find the container with id 96d5ff7d0153c0a45835dc2c26840a8a6bb932ddbcd3ff313f523a1ae23417e8 Jul 17 18:15:00.521781 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:00.521750 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-284e9f2b-29738535-jbzn8"] Jul 17 18:15:00.524983 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:15:00.524957 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfadbbe_8217_4313_a30f_7aef25b102e5.slice/crio-8e67491d4d9d4ea70dea4565b279840f3fd80a24065746eae265bf32d50e169a WatchSource:0}: Error finding container 8e67491d4d9d4ea70dea4565b279840f3fd80a24065746eae265bf32d50e169a: Status 404 returned error can't find the container with id 8e67491d4d9d4ea70dea4565b279840f3fd80a24065746eae265bf32d50e169a Jul 17 18:15:00.784529 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:00.784489 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738535-d9vlp" event={"ID":"d448bbf4-fa79-48e3-9312-b2e64d74478c","Type":"ContainerStarted","Data":"85e6634ef24fc79c85aef49ad57cf2b9a2d406ee0ff16d7a3fc0e28be0be80ec"} Jul 17 18:15:00.784733 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:00.784555 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738535-d9vlp" event={"ID":"d448bbf4-fa79-48e3-9312-b2e64d74478c","Type":"ContainerStarted","Data":"96d5ff7d0153c0a45835dc2c26840a8a6bb932ddbcd3ff313f523a1ae23417e8"} Jul 17 18:15:00.785857 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:00.785834 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-284e9f2b-29738535-jbzn8" event={"ID":"2bfadbbe-8217-4313-a30f-7aef25b102e5","Type":"ContainerStarted","Data":"8e67491d4d9d4ea70dea4565b279840f3fd80a24065746eae265bf32d50e169a"} Jul 17 18:15:00.800648 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:00.800600 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738535-d9vlp" podStartSLOduration=0.800579327 podStartE2EDuration="800.579327ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:15:00.797991335 +0000 UTC m=+1512.385944040" watchObservedRunningTime="2026-07-17 18:15:00.800579327 +0000 UTC m=+1512.388532018" Jul 17 18:15:01.792641 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:01.792605 2574 generic.go:356] "Generic (PLEG): container finished" podID="d448bbf4-fa79-48e3-9312-b2e64d74478c" containerID="85e6634ef24fc79c85aef49ad57cf2b9a2d406ee0ff16d7a3fc0e28be0be80ec" exitCode=0 Jul 17 18:15:01.793108 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:01.792695 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738535-d9vlp" event={"ID":"d448bbf4-fa79-48e3-9312-b2e64d74478c","Type":"ContainerDied","Data":"85e6634ef24fc79c85aef49ad57cf2b9a2d406ee0ff16d7a3fc0e28be0be80ec"} Jul 17 18:15:01.794226 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:01.794201 2574 generic.go:356] "Generic (PLEG): container finished" podID="2bfadbbe-8217-4313-a30f-7aef25b102e5" containerID="72097d26cb5cc2852e9048ab1f44b7d1a8231eab32a9ebe7db84e5868ce96ac1" exitCode=7 Jul 17 18:15:01.794390 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:01.794291 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-284e9f2b-29738535-jbzn8" event={"ID":"2bfadbbe-8217-4313-a30f-7aef25b102e5","Type":"ContainerDied","Data":"72097d26cb5cc2852e9048ab1f44b7d1a8231eab32a9ebe7db84e5868ce96ac1"} Jul 17 18:15:01.794478 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:01.794439 2574 scope.go:117] "RemoveContainer" containerID="72097d26cb5cc2852e9048ab1f44b7d1a8231eab32a9ebe7db84e5868ce96ac1" Jul 17 18:15:02.802010 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:02.801978 2574 generic.go:356] "Generic (PLEG): container finished" podID="2bfadbbe-8217-4313-a30f-7aef25b102e5" containerID="800ef9097a39e633dffc4fdcdfd00bc4555e22143e22fd29173f4440051f6af3" exitCode=7 Jul 17 18:15:02.802473 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:02.802062 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-284e9f2b-29738535-jbzn8" event={"ID":"2bfadbbe-8217-4313-a30f-7aef25b102e5","Type":"ContainerDied","Data":"800ef9097a39e633dffc4fdcdfd00bc4555e22143e22fd29173f4440051f6af3"} Jul 17 18:15:02.802473 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:02.802119 2574 scope.go:117] "RemoveContainer" containerID="72097d26cb5cc2852e9048ab1f44b7d1a8231eab32a9ebe7db84e5868ce96ac1" Jul 17 18:15:02.802473 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:02.802209 2574 scope.go:117] "RemoveContainer" containerID="800ef9097a39e633dffc4fdcdfd00bc4555e22143e22fd29173f4440051f6af3" Jul 17 18:15:02.802911 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:15:02.802474 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-284e9f2b-29738535-jbzn8_odh-ai-gateway-infra(2bfadbbe-8217-4313-a30f-7aef25b102e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-284e9f2b-29738535-jbzn8" podUID="2bfadbbe-8217-4313-a30f-7aef25b102e5" Jul 17 18:15:02.938935 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:02.938899 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738535-d9vlp" Jul 17 18:15:03.808769 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:03.808734 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738535-d9vlp" event={"ID":"d448bbf4-fa79-48e3-9312-b2e64d74478c","Type":"ContainerDied","Data":"96d5ff7d0153c0a45835dc2c26840a8a6bb932ddbcd3ff313f523a1ae23417e8"} Jul 17 18:15:03.809263 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:03.808790 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96d5ff7d0153c0a45835dc2c26840a8a6bb932ddbcd3ff313f523a1ae23417e8" Jul 17 18:15:03.809263 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:03.808744 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738535-d9vlp" Jul 17 18:15:03.810582 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:03.810557 2574 scope.go:117] "RemoveContainer" containerID="800ef9097a39e633dffc4fdcdfd00bc4555e22143e22fd29173f4440051f6af3" Jul 17 18:15:03.810834 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:15:03.810804 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-284e9f2b-29738535-jbzn8_odh-ai-gateway-infra(2bfadbbe-8217-4313-a30f-7aef25b102e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-284e9f2b-29738535-jbzn8" podUID="2bfadbbe-8217-4313-a30f-7aef25b102e5" Jul 17 18:15:05.979580 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:05.979541 2574 scope.go:117] "RemoveContainer" containerID="e03960d8f01ee8239806ceeb5cb9b8a7fb42d3e6f3dc111ca86af3332dc94e50" Jul 17 18:15:06.827679 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:06.827643 2574 generic.go:356] "Generic (PLEG): container finished" podID="1173c07a-0d1f-4e3b-a832-3879cc23b935" containerID="e1818d9b282c39e1cbfee79b96e9554d79afd93eddcb156735b8e0cb576f0a63" exitCode=7 Jul 17 18:15:06.827831 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:06.827715 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h" event={"ID":"1173c07a-0d1f-4e3b-a832-3879cc23b935","Type":"ContainerDied","Data":"e1818d9b282c39e1cbfee79b96e9554d79afd93eddcb156735b8e0cb576f0a63"} Jul 17 18:15:06.827831 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:06.827771 2574 scope.go:117] "RemoveContainer" containerID="e03960d8f01ee8239806ceeb5cb9b8a7fb42d3e6f3dc111ca86af3332dc94e50" Jul 17 18:15:06.828140 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:06.828126 2574 scope.go:117] "RemoveContainer" containerID="e1818d9b282c39e1cbfee79b96e9554d79afd93eddcb156735b8e0cb576f0a63" Jul 17 18:15:06.828417 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:15:06.828388 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h_odh-ai-gateway-infra(1173c07a-0d1f-4e3b-a832-3879cc23b935)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h" podUID="1173c07a-0d1f-4e3b-a832-3879cc23b935" Jul 17 18:15:07.853007 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:07.852971 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h"] Jul 17 18:15:07.997094 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:07.997064 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h" Jul 17 18:15:08.061571 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:08.061532 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1173c07a-0d1f-4e3b-a832-3879cc23b935-maas-api-service-ca\") pod \"1173c07a-0d1f-4e3b-a832-3879cc23b935\" (UID: \"1173c07a-0d1f-4e3b-a832-3879cc23b935\") " Jul 17 18:15:08.061930 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:08.061911 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1173c07a-0d1f-4e3b-a832-3879cc23b935-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1173c07a-0d1f-4e3b-a832-3879cc23b935" (UID: "1173c07a-0d1f-4e3b-a832-3879cc23b935"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:15:08.162813 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:08.162726 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1173c07a-0d1f-4e3b-a832-3879cc23b935-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:15:08.841040 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:08.841008 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h" Jul 17 18:15:08.841246 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:08.841040 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h" event={"ID":"1173c07a-0d1f-4e3b-a832-3879cc23b935","Type":"ContainerDied","Data":"ea33c360998611a98863a7b6c1b448551e9dc4a1610622dd6632c1165a7ff3df"} Jul 17 18:15:08.841246 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:08.841124 2574 scope.go:117] "RemoveContainer" containerID="e1818d9b282c39e1cbfee79b96e9554d79afd93eddcb156735b8e0cb576f0a63" Jul 17 18:15:08.863312 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:08.863282 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h"] Jul 17 18:15:08.868868 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:08.868843 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-vlz4h"] Jul 17 18:15:08.986164 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:08.986127 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1173c07a-0d1f-4e3b-a832-3879cc23b935" path="/var/lib/kubelet/pods/1173c07a-0d1f-4e3b-a832-3879cc23b935/volumes" Jul 17 18:15:15.979868 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:15.979783 2574 scope.go:117] "RemoveContainer" containerID="800ef9097a39e633dffc4fdcdfd00bc4555e22143e22fd29173f4440051f6af3" Jul 17 18:15:16.882956 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:16.882859 2574 generic.go:356] "Generic (PLEG): container finished" podID="2bfadbbe-8217-4313-a30f-7aef25b102e5" containerID="9eee54f73dbd1e3246b2a7320d08683de39bfa2610fc9f1d83b4be458311175f" exitCode=7 Jul 17 18:15:16.882956 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:16.882940 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-284e9f2b-29738535-jbzn8" event={"ID":"2bfadbbe-8217-4313-a30f-7aef25b102e5","Type":"ContainerDied","Data":"9eee54f73dbd1e3246b2a7320d08683de39bfa2610fc9f1d83b4be458311175f"} Jul 17 18:15:16.883181 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:16.883016 2574 scope.go:117] "RemoveContainer" containerID="800ef9097a39e633dffc4fdcdfd00bc4555e22143e22fd29173f4440051f6af3" Jul 17 18:15:16.883342 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:16.883328 2574 scope.go:117] "RemoveContainer" containerID="9eee54f73dbd1e3246b2a7320d08683de39bfa2610fc9f1d83b4be458311175f" Jul 17 18:15:16.883641 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:15:16.883617 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-284e9f2b-29738535-jbzn8_odh-ai-gateway-infra(2bfadbbe-8217-4313-a30f-7aef25b102e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-284e9f2b-29738535-jbzn8" podUID="2bfadbbe-8217-4313-a30f-7aef25b102e5" Jul 17 18:15:17.906681 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:17.906648 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-284e9f2b-29738535-jbzn8"] Jul 17 18:15:18.038946 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:18.038921 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-284e9f2b-29738535-jbzn8" Jul 17 18:15:18.896006 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:18.895976 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-284e9f2b-29738535-jbzn8" Jul 17 18:15:18.896006 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:18.895994 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-284e9f2b-29738535-jbzn8" event={"ID":"2bfadbbe-8217-4313-a30f-7aef25b102e5","Type":"ContainerDied","Data":"8e67491d4d9d4ea70dea4565b279840f3fd80a24065746eae265bf32d50e169a"} Jul 17 18:15:18.896247 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:18.896064 2574 scope.go:117] "RemoveContainer" containerID="9eee54f73dbd1e3246b2a7320d08683de39bfa2610fc9f1d83b4be458311175f" Jul 17 18:15:18.919370 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:18.919306 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-284e9f2b-29738535-jbzn8"] Jul 17 18:15:18.925116 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:18.925090 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-284e9f2b-29738535-jbzn8"] Jul 17 18:15:18.985536 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:18.985500 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bfadbbe-8217-4313-a30f-7aef25b102e5" path="/var/lib/kubelet/pods/2bfadbbe-8217-4313-a30f-7aef25b102e5/volumes" Jul 17 18:15:42.569765 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.569725 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2"] Jul 17 18:15:42.570250 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.570154 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1173c07a-0d1f-4e3b-a832-3879cc23b935" containerName="revoke-keys" Jul 17 18:15:42.570250 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.570165 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1173c07a-0d1f-4e3b-a832-3879cc23b935" containerName="revoke-keys" Jul 17 18:15:42.570250 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.570172 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1173c07a-0d1f-4e3b-a832-3879cc23b935" containerName="revoke-keys" Jul 17 18:15:42.570250 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.570177 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1173c07a-0d1f-4e3b-a832-3879cc23b935" containerName="revoke-keys" Jul 17 18:15:42.570250 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.570183 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bfadbbe-8217-4313-a30f-7aef25b102e5" containerName="cleanup" Jul 17 18:15:42.570250 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.570188 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bfadbbe-8217-4313-a30f-7aef25b102e5" containerName="cleanup" Jul 17 18:15:42.570250 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.570205 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bfadbbe-8217-4313-a30f-7aef25b102e5" containerName="cleanup" Jul 17 18:15:42.570250 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.570216 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bfadbbe-8217-4313-a30f-7aef25b102e5" containerName="cleanup" Jul 17 18:15:42.570250 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.570222 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d448bbf4-fa79-48e3-9312-b2e64d74478c" containerName="cleanup" Jul 17 18:15:42.570250 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.570226 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d448bbf4-fa79-48e3-9312-b2e64d74478c" containerName="cleanup" Jul 17 18:15:42.570250 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.570233 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bfadbbe-8217-4313-a30f-7aef25b102e5" containerName="cleanup" Jul 17 18:15:42.570250 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.570237 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bfadbbe-8217-4313-a30f-7aef25b102e5" containerName="cleanup" Jul 17 18:15:42.570250 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.570251 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1173c07a-0d1f-4e3b-a832-3879cc23b935" containerName="revoke-keys" Jul 17 18:15:42.570250 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.570256 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1173c07a-0d1f-4e3b-a832-3879cc23b935" containerName="revoke-keys" Jul 17 18:15:42.570692 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.570322 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1173c07a-0d1f-4e3b-a832-3879cc23b935" containerName="revoke-keys" Jul 17 18:15:42.570692 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.570329 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d448bbf4-fa79-48e3-9312-b2e64d74478c" containerName="cleanup" Jul 17 18:15:42.570692 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.570334 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bfadbbe-8217-4313-a30f-7aef25b102e5" containerName="cleanup" Jul 17 18:15:42.570692 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.570340 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bfadbbe-8217-4313-a30f-7aef25b102e5" containerName="cleanup" Jul 17 18:15:42.570692 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.570345 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bfadbbe-8217-4313-a30f-7aef25b102e5" containerName="cleanup" Jul 17 18:15:42.570692 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.570349 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1173c07a-0d1f-4e3b-a832-3879cc23b935" containerName="revoke-keys" Jul 17 18:15:42.575106 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.575086 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2" Jul 17 18:15:42.585773 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.582931 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2"] Jul 17 18:15:42.585773 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.583486 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:15:42.585773 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.583814 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dpsm\"" Jul 17 18:15:42.688762 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.688718 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43989a4-ac64-4d98-9943-2ed47c447296-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2\" (UID: \"c43989a4-ac64-4d98-9943-2ed47c447296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2" Jul 17 18:15:42.790429 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.790384 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43989a4-ac64-4d98-9943-2ed47c447296-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2\" (UID: \"c43989a4-ac64-4d98-9943-2ed47c447296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2" Jul 17 18:15:42.790950 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.790928 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43989a4-ac64-4d98-9943-2ed47c447296-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2\" (UID: \"c43989a4-ac64-4d98-9943-2ed47c447296\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2" Jul 17 18:15:42.893599 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:42.893510 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2" Jul 17 18:15:43.034514 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:43.034487 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2"] Jul 17 18:15:43.036827 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:15:43.036799 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43989a4_ac64_4d98_9943_2ed47c447296.slice/crio-5f01a3c65406ee085ca7f80929e8429cc971918c4a9e5a1012b5463add6764ed WatchSource:0}: Error finding container 5f01a3c65406ee085ca7f80929e8429cc971918c4a9e5a1012b5463add6764ed: Status 404 returned error can't find the container with id 5f01a3c65406ee085ca7f80929e8429cc971918c4a9e5a1012b5463add6764ed Jul 17 18:15:44.031993 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:44.031957 2574 generic.go:356] "Generic (PLEG): container finished" podID="c43989a4-ac64-4d98-9943-2ed47c447296" containerID="ef679b657f65c315bb49f82013641e482853f795bc04373c6337eec5003f5a7c" exitCode=7 Jul 17 18:15:44.032421 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:44.032045 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2" event={"ID":"c43989a4-ac64-4d98-9943-2ed47c447296","Type":"ContainerDied","Data":"ef679b657f65c315bb49f82013641e482853f795bc04373c6337eec5003f5a7c"} Jul 17 18:15:44.032421 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:44.032101 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2" event={"ID":"c43989a4-ac64-4d98-9943-2ed47c447296","Type":"ContainerStarted","Data":"5f01a3c65406ee085ca7f80929e8429cc971918c4a9e5a1012b5463add6764ed"} Jul 17 18:15:44.032421 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:44.032193 2574 scope.go:117] "RemoveContainer" containerID="ef679b657f65c315bb49f82013641e482853f795bc04373c6337eec5003f5a7c" Jul 17 18:15:45.039333 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:45.039298 2574 generic.go:356] "Generic (PLEG): container finished" podID="c43989a4-ac64-4d98-9943-2ed47c447296" containerID="57871ec383b431cebdc923675fa2f0482fa4e0e5b319fac109f52f6821bc56b2" exitCode=7 Jul 17 18:15:45.039827 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:45.039381 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2" event={"ID":"c43989a4-ac64-4d98-9943-2ed47c447296","Type":"ContainerDied","Data":"57871ec383b431cebdc923675fa2f0482fa4e0e5b319fac109f52f6821bc56b2"} Jul 17 18:15:45.039827 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:45.039431 2574 scope.go:117] "RemoveContainer" containerID="ef679b657f65c315bb49f82013641e482853f795bc04373c6337eec5003f5a7c" Jul 17 18:15:45.039827 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:45.039617 2574 scope.go:117] "RemoveContainer" containerID="57871ec383b431cebdc923675fa2f0482fa4e0e5b319fac109f52f6821bc56b2" Jul 17 18:15:45.039987 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:15:45.039845 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2_odh-ai-gateway-infra(c43989a4-ac64-4d98-9943-2ed47c447296)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2" podUID="c43989a4-ac64-4d98-9943-2ed47c447296" Jul 17 18:15:46.046222 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:46.046191 2574 scope.go:117] "RemoveContainer" containerID="57871ec383b431cebdc923675fa2f0482fa4e0e5b319fac109f52f6821bc56b2" Jul 17 18:15:46.046628 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:15:46.046415 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2_odh-ai-gateway-infra(c43989a4-ac64-4d98-9943-2ed47c447296)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2" podUID="c43989a4-ac64-4d98-9943-2ed47c447296" Jul 17 18:15:56.979650 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:56.979619 2574 scope.go:117] "RemoveContainer" containerID="57871ec383b431cebdc923675fa2f0482fa4e0e5b319fac109f52f6821bc56b2" Jul 17 18:15:58.109317 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:58.109285 2574 generic.go:356] "Generic (PLEG): container finished" podID="c43989a4-ac64-4d98-9943-2ed47c447296" containerID="db0f2c31dc08a8f40b836b2995c1f9975c171a4eb7e7cd47bee5e43aa599e6be" exitCode=7 Jul 17 18:15:58.109750 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:58.109368 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2" event={"ID":"c43989a4-ac64-4d98-9943-2ed47c447296","Type":"ContainerDied","Data":"db0f2c31dc08a8f40b836b2995c1f9975c171a4eb7e7cd47bee5e43aa599e6be"} Jul 17 18:15:58.109750 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:58.109430 2574 scope.go:117] "RemoveContainer" containerID="57871ec383b431cebdc923675fa2f0482fa4e0e5b319fac109f52f6821bc56b2" Jul 17 18:15:58.109874 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:58.109803 2574 scope.go:117] "RemoveContainer" containerID="db0f2c31dc08a8f40b836b2995c1f9975c171a4eb7e7cd47bee5e43aa599e6be" Jul 17 18:15:58.110085 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:15:58.110066 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2_odh-ai-gateway-infra(c43989a4-ac64-4d98-9943-2ed47c447296)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2" podUID="c43989a4-ac64-4d98-9943-2ed47c447296" Jul 17 18:15:59.134637 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:59.134599 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2"] Jul 17 18:15:59.272924 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:59.272898 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2" Jul 17 18:15:59.454590 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:59.454481 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43989a4-ac64-4d98-9943-2ed47c447296-maas-api-service-ca\") pod \"c43989a4-ac64-4d98-9943-2ed47c447296\" (UID: \"c43989a4-ac64-4d98-9943-2ed47c447296\") " Jul 17 18:15:59.454839 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:59.454813 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c43989a4-ac64-4d98-9943-2ed47c447296-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c43989a4-ac64-4d98-9943-2ed47c447296" (UID: "c43989a4-ac64-4d98-9943-2ed47c447296"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:15:59.555854 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:15:59.555819 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43989a4-ac64-4d98-9943-2ed47c447296-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:16:00.122329 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:00.122298 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2" Jul 17 18:16:00.122529 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:00.122334 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2" event={"ID":"c43989a4-ac64-4d98-9943-2ed47c447296","Type":"ContainerDied","Data":"5f01a3c65406ee085ca7f80929e8429cc971918c4a9e5a1012b5463add6764ed"} Jul 17 18:16:00.122529 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:00.122415 2574 scope.go:117] "RemoveContainer" containerID="db0f2c31dc08a8f40b836b2995c1f9975c171a4eb7e7cd47bee5e43aa599e6be" Jul 17 18:16:00.145311 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:00.145276 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2"] Jul 17 18:16:00.151749 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:00.151723 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zbwh2"] Jul 17 18:16:00.987075 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:00.987039 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c43989a4-ac64-4d98-9943-2ed47c447296" path="/var/lib/kubelet/pods/c43989a4-ac64-4d98-9943-2ed47c447296/volumes" Jul 17 18:16:11.894402 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:11.894342 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:16:14.120927 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:14.120887 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:16:32.611563 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:32.611511 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m"] Jul 17 18:16:32.613692 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:32.612391 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c43989a4-ac64-4d98-9943-2ed47c447296" containerName="revoke-keys" Jul 17 18:16:32.613692 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:32.612411 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c43989a4-ac64-4d98-9943-2ed47c447296" containerName="revoke-keys" Jul 17 18:16:32.613692 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:32.612452 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c43989a4-ac64-4d98-9943-2ed47c447296" containerName="revoke-keys" Jul 17 18:16:32.613692 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:32.612461 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c43989a4-ac64-4d98-9943-2ed47c447296" containerName="revoke-keys" Jul 17 18:16:32.613692 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:32.612626 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c43989a4-ac64-4d98-9943-2ed47c447296" containerName="revoke-keys" Jul 17 18:16:32.613692 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:32.612641 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1173c07a-0d1f-4e3b-a832-3879cc23b935" containerName="revoke-keys" Jul 17 18:16:32.613692 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:32.612658 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c43989a4-ac64-4d98-9943-2ed47c447296" containerName="revoke-keys" Jul 17 18:16:32.616678 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:32.616647 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m" Jul 17 18:16:32.619257 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:32.619233 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dpsm\"" Jul 17 18:16:32.619586 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:32.619523 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:16:32.619676 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:32.619588 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m"] Jul 17 18:16:32.775428 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:32.775380 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc07e4f-f4dd-462b-92c9-178ee18bb776-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m\" (UID: \"fbc07e4f-f4dd-462b-92c9-178ee18bb776\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m" Jul 17 18:16:32.876645 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:32.876529 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc07e4f-f4dd-462b-92c9-178ee18bb776-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m\" (UID: \"fbc07e4f-f4dd-462b-92c9-178ee18bb776\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m" Jul 17 18:16:32.877221 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:32.877195 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc07e4f-f4dd-462b-92c9-178ee18bb776-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m\" (UID: \"fbc07e4f-f4dd-462b-92c9-178ee18bb776\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m" Jul 17 18:16:32.929518 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:32.929474 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m" Jul 17 18:16:33.072393 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:33.072309 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m"] Jul 17 18:16:33.074772 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:16:33.074743 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc07e4f_f4dd_462b_92c9_178ee18bb776.slice/crio-8444d20bc991509d6a59d86d69c90861044a5470a37756f0b203a021c9c031d3 WatchSource:0}: Error finding container 8444d20bc991509d6a59d86d69c90861044a5470a37756f0b203a021c9c031d3: Status 404 returned error can't find the container with id 8444d20bc991509d6a59d86d69c90861044a5470a37756f0b203a021c9c031d3 Jul 17 18:16:33.286347 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:33.286311 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m" event={"ID":"fbc07e4f-f4dd-462b-92c9-178ee18bb776","Type":"ContainerStarted","Data":"8444d20bc991509d6a59d86d69c90861044a5470a37756f0b203a021c9c031d3"} Jul 17 18:16:34.293321 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:34.293281 2574 generic.go:356] "Generic (PLEG): container finished" podID="fbc07e4f-f4dd-462b-92c9-178ee18bb776" containerID="021edec8fe260f413f9021e3696938547d7f512d7ccda957360d867aaaf92a7f" exitCode=7 Jul 17 18:16:34.293753 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:34.293384 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m" event={"ID":"fbc07e4f-f4dd-462b-92c9-178ee18bb776","Type":"ContainerDied","Data":"021edec8fe260f413f9021e3696938547d7f512d7ccda957360d867aaaf92a7f"} Jul 17 18:16:34.293753 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:34.293547 2574 scope.go:117] "RemoveContainer" containerID="021edec8fe260f413f9021e3696938547d7f512d7ccda957360d867aaaf92a7f" Jul 17 18:16:35.303220 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:35.303185 2574 generic.go:356] "Generic (PLEG): container finished" podID="fbc07e4f-f4dd-462b-92c9-178ee18bb776" containerID="6a6a714b3989ecc4e148f348d1f5f3c1d3a0c888b47ee6a4e48874f46f4c5a93" exitCode=7 Jul 17 18:16:35.303734 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:35.303268 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m" event={"ID":"fbc07e4f-f4dd-462b-92c9-178ee18bb776","Type":"ContainerDied","Data":"6a6a714b3989ecc4e148f348d1f5f3c1d3a0c888b47ee6a4e48874f46f4c5a93"} Jul 17 18:16:35.303734 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:35.303336 2574 scope.go:117] "RemoveContainer" containerID="021edec8fe260f413f9021e3696938547d7f512d7ccda957360d867aaaf92a7f" Jul 17 18:16:35.303734 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:35.303561 2574 scope.go:117] "RemoveContainer" containerID="6a6a714b3989ecc4e148f348d1f5f3c1d3a0c888b47ee6a4e48874f46f4c5a93" Jul 17 18:16:35.303871 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:16:35.303756 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m_odh-ai-gateway-infra(fbc07e4f-f4dd-462b-92c9-178ee18bb776)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m" podUID="fbc07e4f-f4dd-462b-92c9-178ee18bb776" Jul 17 18:16:36.309784 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:36.309757 2574 scope.go:117] "RemoveContainer" containerID="6a6a714b3989ecc4e148f348d1f5f3c1d3a0c888b47ee6a4e48874f46f4c5a93" Jul 17 18:16:36.310153 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:16:36.309963 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m_odh-ai-gateway-infra(fbc07e4f-f4dd-462b-92c9-178ee18bb776)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m" podUID="fbc07e4f-f4dd-462b-92c9-178ee18bb776" Jul 17 18:16:50.980471 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:50.980438 2574 scope.go:117] "RemoveContainer" containerID="6a6a714b3989ecc4e148f348d1f5f3c1d3a0c888b47ee6a4e48874f46f4c5a93" Jul 17 18:16:51.388675 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:51.388636 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m" event={"ID":"fbc07e4f-f4dd-462b-92c9-178ee18bb776","Type":"ContainerStarted","Data":"a9cc9e135b938e3d49536e66f9f21bb49033a9d15054d5fb17e50230a2d81705"} Jul 17 18:16:51.406349 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:51.406295 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m" podStartSLOduration=19.406279521 podStartE2EDuration="19.406279521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:16:51.405412509 +0000 UTC m=+1622.993365200" watchObservedRunningTime="2026-07-17 18:16:51.406279521 +0000 UTC m=+1622.994232211" Jul 17 18:16:52.395973 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:52.395935 2574 generic.go:356] "Generic (PLEG): container finished" podID="fbc07e4f-f4dd-462b-92c9-178ee18bb776" containerID="a9cc9e135b938e3d49536e66f9f21bb49033a9d15054d5fb17e50230a2d81705" exitCode=7 Jul 17 18:16:52.396481 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:52.396008 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m" event={"ID":"fbc07e4f-f4dd-462b-92c9-178ee18bb776","Type":"ContainerDied","Data":"a9cc9e135b938e3d49536e66f9f21bb49033a9d15054d5fb17e50230a2d81705"} Jul 17 18:16:52.396481 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:52.396068 2574 scope.go:117] "RemoveContainer" containerID="6a6a714b3989ecc4e148f348d1f5f3c1d3a0c888b47ee6a4e48874f46f4c5a93" Jul 17 18:16:52.396481 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:52.396412 2574 scope.go:117] "RemoveContainer" containerID="a9cc9e135b938e3d49536e66f9f21bb49033a9d15054d5fb17e50230a2d81705" Jul 17 18:16:52.396694 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:16:52.396673 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m_odh-ai-gateway-infra(fbc07e4f-f4dd-462b-92c9-178ee18bb776)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m" podUID="fbc07e4f-f4dd-462b-92c9-178ee18bb776" Jul 17 18:16:52.417503 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:52.417467 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m"] Jul 17 18:16:53.541808 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:53.541779 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m" Jul 17 18:16:53.677009 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:53.676903 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc07e4f-f4dd-462b-92c9-178ee18bb776-maas-api-service-ca\") pod \"fbc07e4f-f4dd-462b-92c9-178ee18bb776\" (UID: \"fbc07e4f-f4dd-462b-92c9-178ee18bb776\") " Jul 17 18:16:53.677300 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:53.677274 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbc07e4f-f4dd-462b-92c9-178ee18bb776-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fbc07e4f-f4dd-462b-92c9-178ee18bb776" (UID: "fbc07e4f-f4dd-462b-92c9-178ee18bb776"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:16:53.778506 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:53.778453 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbc07e4f-f4dd-462b-92c9-178ee18bb776-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:16:54.409344 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:54.409313 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m" Jul 17 18:16:54.409588 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:54.409368 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m" event={"ID":"fbc07e4f-f4dd-462b-92c9-178ee18bb776","Type":"ContainerDied","Data":"8444d20bc991509d6a59d86d69c90861044a5470a37756f0b203a021c9c031d3"} Jul 17 18:16:54.409588 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:54.409435 2574 scope.go:117] "RemoveContainer" containerID="a9cc9e135b938e3d49536e66f9f21bb49033a9d15054d5fb17e50230a2d81705" Jul 17 18:16:54.432621 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:54.432591 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m"] Jul 17 18:16:54.438268 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:54.438240 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-cvh7m"] Jul 17 18:16:54.985848 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:16:54.985813 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbc07e4f-f4dd-462b-92c9-178ee18bb776" path="/var/lib/kubelet/pods/fbc07e4f-f4dd-462b-92c9-178ee18bb776/volumes" Jul 17 18:17:32.649182 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:32.649146 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2"] Jul 17 18:17:32.649674 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:32.649604 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c43989a4-ac64-4d98-9943-2ed47c447296" containerName="revoke-keys" Jul 17 18:17:32.649674 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:32.649617 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c43989a4-ac64-4d98-9943-2ed47c447296" containerName="revoke-keys" Jul 17 18:17:32.649674 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:32.649625 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbc07e4f-f4dd-462b-92c9-178ee18bb776" containerName="revoke-keys" Jul 17 18:17:32.649674 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:32.649630 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbc07e4f-f4dd-462b-92c9-178ee18bb776" containerName="revoke-keys" Jul 17 18:17:32.649674 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:32.649644 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbc07e4f-f4dd-462b-92c9-178ee18bb776" containerName="revoke-keys" Jul 17 18:17:32.649674 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:32.649651 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbc07e4f-f4dd-462b-92c9-178ee18bb776" containerName="revoke-keys" Jul 17 18:17:32.649674 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:32.649658 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbc07e4f-f4dd-462b-92c9-178ee18bb776" containerName="revoke-keys" Jul 17 18:17:32.649674 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:32.649663 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbc07e4f-f4dd-462b-92c9-178ee18bb776" containerName="revoke-keys" Jul 17 18:17:32.649925 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:32.649720 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbc07e4f-f4dd-462b-92c9-178ee18bb776" containerName="revoke-keys" Jul 17 18:17:32.649925 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:32.649728 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbc07e4f-f4dd-462b-92c9-178ee18bb776" containerName="revoke-keys" Jul 17 18:17:32.649925 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:32.649735 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c43989a4-ac64-4d98-9943-2ed47c447296" containerName="revoke-keys" Jul 17 18:17:32.652864 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:32.652848 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2" Jul 17 18:17:32.655331 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:32.655304 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:17:32.655331 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:32.655326 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dpsm\"" Jul 17 18:17:32.659620 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:32.659594 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2"] Jul 17 18:17:32.759063 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:32.759020 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78704988-8a69-478f-9cb0-85c9900c581a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2\" (UID: \"78704988-8a69-478f-9cb0-85c9900c581a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2" Jul 17 18:17:32.860286 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:32.860251 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78704988-8a69-478f-9cb0-85c9900c581a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2\" (UID: \"78704988-8a69-478f-9cb0-85c9900c581a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2" Jul 17 18:17:32.860937 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:32.860916 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78704988-8a69-478f-9cb0-85c9900c581a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2\" (UID: \"78704988-8a69-478f-9cb0-85c9900c581a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2" Jul 17 18:17:32.966114 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:32.966088 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2" Jul 17 18:17:33.106814 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:33.106785 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2"] Jul 17 18:17:33.109043 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:17:33.109011 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78704988_8a69_478f_9cb0_85c9900c581a.slice/crio-3550c333a41cae6053e62efd36b6338e26db45e401d50ea6faa299c0a3065bf9 WatchSource:0}: Error finding container 3550c333a41cae6053e62efd36b6338e26db45e401d50ea6faa299c0a3065bf9: Status 404 returned error can't find the container with id 3550c333a41cae6053e62efd36b6338e26db45e401d50ea6faa299c0a3065bf9 Jul 17 18:17:33.613460 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:33.613342 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2" event={"ID":"78704988-8a69-478f-9cb0-85c9900c581a","Type":"ContainerStarted","Data":"b218c2a6bea7373019b1edcc0e00a8138952cc8342db2c3548180b667b9f67f5"} Jul 17 18:17:33.613460 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:33.613421 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2" event={"ID":"78704988-8a69-478f-9cb0-85c9900c581a","Type":"ContainerStarted","Data":"3550c333a41cae6053e62efd36b6338e26db45e401d50ea6faa299c0a3065bf9"} Jul 17 18:17:33.631300 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:33.631250 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2" podStartSLOduration=1.631232 podStartE2EDuration="1.631232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:17:33.630849386 +0000 UTC m=+1665.218802076" watchObservedRunningTime="2026-07-17 18:17:33.631232 +0000 UTC m=+1665.219184693" Jul 17 18:17:34.619525 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:34.619493 2574 generic.go:356] "Generic (PLEG): container finished" podID="78704988-8a69-478f-9cb0-85c9900c581a" containerID="b218c2a6bea7373019b1edcc0e00a8138952cc8342db2c3548180b667b9f67f5" exitCode=7 Jul 17 18:17:34.620033 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:34.619585 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2" event={"ID":"78704988-8a69-478f-9cb0-85c9900c581a","Type":"ContainerDied","Data":"b218c2a6bea7373019b1edcc0e00a8138952cc8342db2c3548180b667b9f67f5"} Jul 17 18:17:34.620033 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:34.619783 2574 scope.go:117] "RemoveContainer" containerID="b218c2a6bea7373019b1edcc0e00a8138952cc8342db2c3548180b667b9f67f5" Jul 17 18:17:35.627016 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:35.626979 2574 generic.go:356] "Generic (PLEG): container finished" podID="78704988-8a69-478f-9cb0-85c9900c581a" containerID="1f43157ce1d81628e7fcefb6a54491156b269ac5b2f89f0298061f588411ca84" exitCode=7 Jul 17 18:17:35.627444 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:35.627061 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2" event={"ID":"78704988-8a69-478f-9cb0-85c9900c581a","Type":"ContainerDied","Data":"1f43157ce1d81628e7fcefb6a54491156b269ac5b2f89f0298061f588411ca84"} Jul 17 18:17:35.627444 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:35.627152 2574 scope.go:117] "RemoveContainer" containerID="b218c2a6bea7373019b1edcc0e00a8138952cc8342db2c3548180b667b9f67f5" Jul 17 18:17:35.627444 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:35.627326 2574 scope.go:117] "RemoveContainer" containerID="1f43157ce1d81628e7fcefb6a54491156b269ac5b2f89f0298061f588411ca84" Jul 17 18:17:35.627619 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:17:35.627603 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2_odh-ai-gateway-infra(78704988-8a69-478f-9cb0-85c9900c581a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2" podUID="78704988-8a69-478f-9cb0-85c9900c581a" Jul 17 18:17:36.633705 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:36.633679 2574 scope.go:117] "RemoveContainer" containerID="1f43157ce1d81628e7fcefb6a54491156b269ac5b2f89f0298061f588411ca84" Jul 17 18:17:36.634165 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:17:36.633873 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2_odh-ai-gateway-infra(78704988-8a69-478f-9cb0-85c9900c581a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2" podUID="78704988-8a69-478f-9cb0-85c9900c581a" Jul 17 18:17:49.979657 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:49.979618 2574 scope.go:117] "RemoveContainer" containerID="1f43157ce1d81628e7fcefb6a54491156b269ac5b2f89f0298061f588411ca84" Jul 17 18:17:50.706204 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:50.706172 2574 generic.go:356] "Generic (PLEG): container finished" podID="78704988-8a69-478f-9cb0-85c9900c581a" containerID="6a6539474e2691484defce2191add384f1fa12b6bb8c071333dd275bb5444cb8" exitCode=7 Jul 17 18:17:50.706388 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:50.706216 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2" event={"ID":"78704988-8a69-478f-9cb0-85c9900c581a","Type":"ContainerDied","Data":"6a6539474e2691484defce2191add384f1fa12b6bb8c071333dd275bb5444cb8"} Jul 17 18:17:50.706388 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:50.706291 2574 scope.go:117] "RemoveContainer" containerID="1f43157ce1d81628e7fcefb6a54491156b269ac5b2f89f0298061f588411ca84" Jul 17 18:17:50.706702 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:50.706684 2574 scope.go:117] "RemoveContainer" containerID="6a6539474e2691484defce2191add384f1fa12b6bb8c071333dd275bb5444cb8" Jul 17 18:17:50.706918 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:17:50.706897 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2_odh-ai-gateway-infra(78704988-8a69-478f-9cb0-85c9900c581a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2" podUID="78704988-8a69-478f-9cb0-85c9900c581a" Jul 17 18:17:51.730197 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:51.730156 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2"] Jul 17 18:17:51.865580 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:51.865554 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2" Jul 17 18:17:51.942949 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:51.942915 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78704988-8a69-478f-9cb0-85c9900c581a-maas-api-service-ca\") pod \"78704988-8a69-478f-9cb0-85c9900c581a\" (UID: \"78704988-8a69-478f-9cb0-85c9900c581a\") " Jul 17 18:17:51.943287 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:51.943264 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78704988-8a69-478f-9cb0-85c9900c581a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78704988-8a69-478f-9cb0-85c9900c581a" (UID: "78704988-8a69-478f-9cb0-85c9900c581a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:17:52.044165 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:52.044060 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78704988-8a69-478f-9cb0-85c9900c581a-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:17:52.699756 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:52.699721 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:52.719461 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:52.719428 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2" Jul 17 18:17:52.719461 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:52.719454 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2" event={"ID":"78704988-8a69-478f-9cb0-85c9900c581a","Type":"ContainerDied","Data":"3550c333a41cae6053e62efd36b6338e26db45e401d50ea6faa299c0a3065bf9"} Jul 17 18:17:52.719672 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:52.719518 2574 scope.go:117] "RemoveContainer" containerID="6a6539474e2691484defce2191add384f1fa12b6bb8c071333dd275bb5444cb8" Jul 17 18:17:52.742534 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:52.742507 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2"] Jul 17 18:17:52.752681 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:52.752649 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-fnnj2"] Jul 17 18:17:52.784781 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:52.784741 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:52.986292 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:52.986256 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78704988-8a69-478f-9cb0-85c9900c581a" path="/var/lib/kubelet/pods/78704988-8a69-478f-9cb0-85c9900c581a/volumes" Jul 17 18:17:58.100612 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:58.100575 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:58.689566 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:58.689518 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:59.629668 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:59.629634 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:17:59.728431 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:17:59.728398 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:18:04.249940 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:04.249904 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:18:04.349999 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:04.349963 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:18:04.572243 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:04.572146 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:18:04.672523 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:04.672493 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:18:04.767021 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:04.766983 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:18:05.965119 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:05.965085 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:18:06.082910 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:06.082877 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:18:06.246684 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:06.246596 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:18:06.550053 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:06.549957 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:18:06.674093 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:06.674064 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:18:10.499078 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:10.498982 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-jfr25"] Jul 17 18:18:10.499590 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:10.499463 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78704988-8a69-478f-9cb0-85c9900c581a" containerName="revoke-keys" Jul 17 18:18:10.499590 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:10.499478 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="78704988-8a69-478f-9cb0-85c9900c581a" containerName="revoke-keys" Jul 17 18:18:10.499590 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:10.499494 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78704988-8a69-478f-9cb0-85c9900c581a" containerName="revoke-keys" Jul 17 18:18:10.499590 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:10.499500 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="78704988-8a69-478f-9cb0-85c9900c581a" containerName="revoke-keys" Jul 17 18:18:10.499590 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:10.499520 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78704988-8a69-478f-9cb0-85c9900c581a" containerName="revoke-keys" Jul 17 18:18:10.499590 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:10.499525 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="78704988-8a69-478f-9cb0-85c9900c581a" containerName="revoke-keys" Jul 17 18:18:10.499590 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:10.499591 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="78704988-8a69-478f-9cb0-85c9900c581a" containerName="revoke-keys" Jul 17 18:18:10.499844 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:10.499603 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="78704988-8a69-478f-9cb0-85c9900c581a" containerName="revoke-keys" Jul 17 18:18:10.499844 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:10.499610 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbc07e4f-f4dd-462b-92c9-178ee18bb776" containerName="revoke-keys" Jul 17 18:18:10.504529 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:10.504510 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-jfr25" Jul 17 18:18:10.508369 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:10.507942 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:18:10.508369 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:10.507996 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dpsm\"" Jul 17 18:18:10.509622 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:10.509597 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-jfr25"] Jul 17 18:18:10.618526 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:10.618490 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-jfr25\" (UID: \"8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-jfr25" Jul 17 18:18:10.720060 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:10.720021 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-jfr25\" (UID: \"8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-jfr25" Jul 17 18:18:10.720673 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:10.720652 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-jfr25\" (UID: \"8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-jfr25" Jul 17 18:18:10.817827 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:10.817742 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-jfr25" Jul 17 18:18:10.956713 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:10.956684 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-jfr25"] Jul 17 18:18:10.959891 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:18:10.959865 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc6b2b1_9bd2_4ffd_b966_af0615f6fa90.slice/crio-4912e3bc8d2eaf2a60aae0d7a6d2711fbad001e350a9cb44b4a79dcbfad8e097 WatchSource:0}: Error finding container 4912e3bc8d2eaf2a60aae0d7a6d2711fbad001e350a9cb44b4a79dcbfad8e097: Status 404 returned error can't find the container with id 4912e3bc8d2eaf2a60aae0d7a6d2711fbad001e350a9cb44b4a79dcbfad8e097 Jul 17 18:18:11.821657 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:11.821619 2574 generic.go:356] "Generic (PLEG): container finished" podID="8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90" containerID="50d59c38d78076812accf042f8bcb179c5cc66f314a0468eba7a3563db30d708" exitCode=7 Jul 17 18:18:11.822072 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:11.821718 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-jfr25" event={"ID":"8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90","Type":"ContainerDied","Data":"50d59c38d78076812accf042f8bcb179c5cc66f314a0468eba7a3563db30d708"} Jul 17 18:18:11.822072 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:11.821784 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-jfr25" event={"ID":"8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90","Type":"ContainerStarted","Data":"4912e3bc8d2eaf2a60aae0d7a6d2711fbad001e350a9cb44b4a79dcbfad8e097"} Jul 17 18:18:11.822072 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:11.821902 2574 scope.go:117] "RemoveContainer" containerID="50d59c38d78076812accf042f8bcb179c5cc66f314a0468eba7a3563db30d708" Jul 17 18:18:12.828486 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:12.828453 2574 generic.go:356] "Generic (PLEG): container finished" podID="8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90" containerID="06a7288bab1017059efc5f2008669adb82bef2095aaf02634de1ee0a23bb23b7" exitCode=7 Jul 17 18:18:12.828989 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:12.828537 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-jfr25" event={"ID":"8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90","Type":"ContainerDied","Data":"06a7288bab1017059efc5f2008669adb82bef2095aaf02634de1ee0a23bb23b7"} Jul 17 18:18:12.828989 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:12.828613 2574 scope.go:117] "RemoveContainer" containerID="50d59c38d78076812accf042f8bcb179c5cc66f314a0468eba7a3563db30d708" Jul 17 18:18:12.828989 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:12.828853 2574 scope.go:117] "RemoveContainer" containerID="06a7288bab1017059efc5f2008669adb82bef2095aaf02634de1ee0a23bb23b7" Jul 17 18:18:12.829163 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:18:12.829058 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-jfr25_odh-ai-gateway-infra(8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-jfr25" podUID="8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90" Jul 17 18:18:13.834993 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:13.834967 2574 scope.go:117] "RemoveContainer" containerID="06a7288bab1017059efc5f2008669adb82bef2095aaf02634de1ee0a23bb23b7" Jul 17 18:18:13.835422 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:18:13.835160 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-jfr25_odh-ai-gateway-infra(8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-jfr25" podUID="8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90" Jul 17 18:18:14.764746 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:14.764709 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:18:14.863940 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:14.863896 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:18:18.412625 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:18.412585 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:18:25.979737 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:25.979700 2574 scope.go:117] "RemoveContainer" containerID="06a7288bab1017059efc5f2008669adb82bef2095aaf02634de1ee0a23bb23b7" Jul 17 18:18:26.904672 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:26.904578 2574 generic.go:356] "Generic (PLEG): container finished" podID="8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90" containerID="bdfa748a412f0fe0e171a000e6476f657b8e8fd65e475f027ade1ce0fbaeda7b" exitCode=7 Jul 17 18:18:26.904672 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:26.904649 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-jfr25" event={"ID":"8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90","Type":"ContainerDied","Data":"bdfa748a412f0fe0e171a000e6476f657b8e8fd65e475f027ade1ce0fbaeda7b"} Jul 17 18:18:26.904857 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:26.904743 2574 scope.go:117] "RemoveContainer" containerID="06a7288bab1017059efc5f2008669adb82bef2095aaf02634de1ee0a23bb23b7" Jul 17 18:18:26.905018 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:26.904997 2574 scope.go:117] "RemoveContainer" containerID="bdfa748a412f0fe0e171a000e6476f657b8e8fd65e475f027ade1ce0fbaeda7b" Jul 17 18:18:26.905292 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:18:26.905271 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-jfr25_odh-ai-gateway-infra(8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-jfr25" podUID="8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90" Jul 17 18:18:27.929296 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:27.929263 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-jfr25"] Jul 17 18:18:28.070575 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:28.070550 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-jfr25" Jul 17 18:18:28.192828 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:28.192723 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90-maas-api-service-ca\") pod \"8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90\" (UID: \"8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90\") " Jul 17 18:18:28.193084 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:28.193062 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90" (UID: "8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:18:28.293720 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:28.293682 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:18:28.917881 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:28.917851 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-jfr25" Jul 17 18:18:28.918098 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:28.917847 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-jfr25" event={"ID":"8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90","Type":"ContainerDied","Data":"4912e3bc8d2eaf2a60aae0d7a6d2711fbad001e350a9cb44b4a79dcbfad8e097"} Jul 17 18:18:28.918098 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:28.917991 2574 scope.go:117] "RemoveContainer" containerID="bdfa748a412f0fe0e171a000e6476f657b8e8fd65e475f027ade1ce0fbaeda7b" Jul 17 18:18:28.940745 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:28.940715 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-jfr25"] Jul 17 18:18:28.946801 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:28.946769 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-jfr25"] Jul 17 18:18:28.985957 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:28.985926 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90" path="/var/lib/kubelet/pods/8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90/volumes" Jul 17 18:18:32.683244 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:32.683206 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8"] Jul 17 18:18:32.683652 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:32.683600 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90" containerName="revoke-keys" Jul 17 18:18:32.683652 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:32.683611 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90" containerName="revoke-keys" Jul 17 18:18:32.683652 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:32.683635 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90" containerName="revoke-keys" Jul 17 18:18:32.683652 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:32.683640 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90" containerName="revoke-keys" Jul 17 18:18:32.683789 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:32.683697 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="78704988-8a69-478f-9cb0-85c9900c581a" containerName="revoke-keys" Jul 17 18:18:32.683789 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:32.683706 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90" containerName="revoke-keys" Jul 17 18:18:32.683789 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:32.683715 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90" containerName="revoke-keys" Jul 17 18:18:32.688198 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:32.688180 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8" Jul 17 18:18:32.691918 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:32.691742 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dpsm\"" Jul 17 18:18:32.691918 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:32.691769 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:18:32.696115 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:32.696084 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8"] Jul 17 18:18:32.727705 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:32.727669 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6cf51b-5947-40dd-8afa-92e5943d4e2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8\" (UID: \"8a6cf51b-5947-40dd-8afa-92e5943d4e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8" Jul 17 18:18:32.828538 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:32.828503 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6cf51b-5947-40dd-8afa-92e5943d4e2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8\" (UID: \"8a6cf51b-5947-40dd-8afa-92e5943d4e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8" Jul 17 18:18:32.829178 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:32.829143 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6cf51b-5947-40dd-8afa-92e5943d4e2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8\" (UID: \"8a6cf51b-5947-40dd-8afa-92e5943d4e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8" Jul 17 18:18:33.007715 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:33.007678 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8" Jul 17 18:18:33.145838 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:33.145812 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8"] Jul 17 18:18:33.147708 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:18:33.147675 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6cf51b_5947_40dd_8afa_92e5943d4e2e.slice/crio-ed121face1db44babedab260a493b58cc3866cc0552209864bb0cd562d870da1 WatchSource:0}: Error finding container ed121face1db44babedab260a493b58cc3866cc0552209864bb0cd562d870da1: Status 404 returned error can't find the container with id ed121face1db44babedab260a493b58cc3866cc0552209864bb0cd562d870da1 Jul 17 18:18:33.945342 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:33.945307 2574 generic.go:356] "Generic (PLEG): container finished" podID="8a6cf51b-5947-40dd-8afa-92e5943d4e2e" containerID="1912c4cf35e3c19c1a156c2f0464e014bf5ab763e1c776ab40b54a892bf891c1" exitCode=6 Jul 17 18:18:33.945768 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:33.945391 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8" event={"ID":"8a6cf51b-5947-40dd-8afa-92e5943d4e2e","Type":"ContainerDied","Data":"1912c4cf35e3c19c1a156c2f0464e014bf5ab763e1c776ab40b54a892bf891c1"} Jul 17 18:18:33.945768 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:33.945446 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8" event={"ID":"8a6cf51b-5947-40dd-8afa-92e5943d4e2e","Type":"ContainerStarted","Data":"ed121face1db44babedab260a493b58cc3866cc0552209864bb0cd562d870da1"} Jul 17 18:18:33.945768 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:33.945649 2574 scope.go:117] "RemoveContainer" containerID="1912c4cf35e3c19c1a156c2f0464e014bf5ab763e1c776ab40b54a892bf891c1" Jul 17 18:18:34.953735 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:34.953642 2574 generic.go:356] "Generic (PLEG): container finished" podID="8a6cf51b-5947-40dd-8afa-92e5943d4e2e" containerID="8b6766cac770ed90a33b98a000b16a8b7fde3b494b3515dfe7d39aa1db0f4b77" exitCode=6 Jul 17 18:18:34.953735 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:34.953718 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8" event={"ID":"8a6cf51b-5947-40dd-8afa-92e5943d4e2e","Type":"ContainerDied","Data":"8b6766cac770ed90a33b98a000b16a8b7fde3b494b3515dfe7d39aa1db0f4b77"} Jul 17 18:18:34.954239 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:34.953787 2574 scope.go:117] "RemoveContainer" containerID="1912c4cf35e3c19c1a156c2f0464e014bf5ab763e1c776ab40b54a892bf891c1" Jul 17 18:18:34.954239 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:34.953977 2574 scope.go:117] "RemoveContainer" containerID="8b6766cac770ed90a33b98a000b16a8b7fde3b494b3515dfe7d39aa1db0f4b77" Jul 17 18:18:34.954318 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:18:34.954248 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8_odh-ai-gateway-infra(8a6cf51b-5947-40dd-8afa-92e5943d4e2e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8" podUID="8a6cf51b-5947-40dd-8afa-92e5943d4e2e" Jul 17 18:18:35.960278 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:35.960251 2574 scope.go:117] "RemoveContainer" containerID="8b6766cac770ed90a33b98a000b16a8b7fde3b494b3515dfe7d39aa1db0f4b77" Jul 17 18:18:35.960681 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:18:35.960487 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8_odh-ai-gateway-infra(8a6cf51b-5947-40dd-8afa-92e5943d4e2e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8" podUID="8a6cf51b-5947-40dd-8afa-92e5943d4e2e" Jul 17 18:18:47.979886 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:47.979854 2574 scope.go:117] "RemoveContainer" containerID="8b6766cac770ed90a33b98a000b16a8b7fde3b494b3515dfe7d39aa1db0f4b77" Jul 17 18:18:49.030048 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:49.030013 2574 generic.go:356] "Generic (PLEG): container finished" podID="8a6cf51b-5947-40dd-8afa-92e5943d4e2e" containerID="74f96b17670b7c385dc536b0c8fd90dff83f1724949fb1ceab784ab62091ceea" exitCode=6 Jul 17 18:18:49.030513 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:49.030092 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8" event={"ID":"8a6cf51b-5947-40dd-8afa-92e5943d4e2e","Type":"ContainerDied","Data":"74f96b17670b7c385dc536b0c8fd90dff83f1724949fb1ceab784ab62091ceea"} Jul 17 18:18:49.030513 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:49.030168 2574 scope.go:117] "RemoveContainer" containerID="8b6766cac770ed90a33b98a000b16a8b7fde3b494b3515dfe7d39aa1db0f4b77" Jul 17 18:18:49.030622 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:49.030521 2574 scope.go:117] "RemoveContainer" containerID="74f96b17670b7c385dc536b0c8fd90dff83f1724949fb1ceab784ab62091ceea" Jul 17 18:18:49.030757 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:18:49.030734 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8_odh-ai-gateway-infra(8a6cf51b-5947-40dd-8afa-92e5943d4e2e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8" podUID="8a6cf51b-5947-40dd-8afa-92e5943d4e2e" Jul 17 18:18:50.054769 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:50.054739 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8"] Jul 17 18:18:50.188384 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:50.188339 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8" Jul 17 18:18:50.291330 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:50.291292 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6cf51b-5947-40dd-8afa-92e5943d4e2e-maas-api-service-ca\") pod \"8a6cf51b-5947-40dd-8afa-92e5943d4e2e\" (UID: \"8a6cf51b-5947-40dd-8afa-92e5943d4e2e\") " Jul 17 18:18:50.291747 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:50.291719 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a6cf51b-5947-40dd-8afa-92e5943d4e2e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8a6cf51b-5947-40dd-8afa-92e5943d4e2e" (UID: "8a6cf51b-5947-40dd-8afa-92e5943d4e2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:18:50.393093 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:50.393004 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a6cf51b-5947-40dd-8afa-92e5943d4e2e-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:18:51.042807 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:51.042776 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8" Jul 17 18:18:51.042807 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:51.042786 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8" event={"ID":"8a6cf51b-5947-40dd-8afa-92e5943d4e2e","Type":"ContainerDied","Data":"ed121face1db44babedab260a493b58cc3866cc0552209864bb0cd562d870da1"} Jul 17 18:18:51.043061 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:51.042847 2574 scope.go:117] "RemoveContainer" containerID="74f96b17670b7c385dc536b0c8fd90dff83f1724949fb1ceab784ab62091ceea" Jul 17 18:18:51.058800 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:51.058774 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8"] Jul 17 18:18:51.064887 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:51.064852 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xw2j8"] Jul 17 18:18:52.985773 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:18:52.985737 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a6cf51b-5947-40dd-8afa-92e5943d4e2e" path="/var/lib/kubelet/pods/8a6cf51b-5947-40dd-8afa-92e5943d4e2e/volumes" Jul 17 18:19:00.534041 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:00.534002 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9g22g"] Jul 17 18:19:00.534441 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:00.534410 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90" containerName="revoke-keys" Jul 17 18:19:00.534441 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:00.534421 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90" containerName="revoke-keys" Jul 17 18:19:00.534441 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:00.534430 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a6cf51b-5947-40dd-8afa-92e5943d4e2e" containerName="revoke-keys" Jul 17 18:19:00.534441 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:00.534435 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a6cf51b-5947-40dd-8afa-92e5943d4e2e" containerName="revoke-keys" Jul 17 18:19:00.534587 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:00.534447 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a6cf51b-5947-40dd-8afa-92e5943d4e2e" containerName="revoke-keys" Jul 17 18:19:00.534587 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:00.534453 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a6cf51b-5947-40dd-8afa-92e5943d4e2e" containerName="revoke-keys" Jul 17 18:19:00.534587 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:00.534461 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a6cf51b-5947-40dd-8afa-92e5943d4e2e" containerName="revoke-keys" Jul 17 18:19:00.534587 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:00.534465 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a6cf51b-5947-40dd-8afa-92e5943d4e2e" containerName="revoke-keys" Jul 17 18:19:00.534587 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:00.534527 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a6cf51b-5947-40dd-8afa-92e5943d4e2e" containerName="revoke-keys" Jul 17 18:19:00.534587 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:00.534536 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cc6b2b1-9bd2-4ffd-b966-af0615f6fa90" containerName="revoke-keys" Jul 17 18:19:00.534587 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:00.534543 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a6cf51b-5947-40dd-8afa-92e5943d4e2e" containerName="revoke-keys" Jul 17 18:19:00.539063 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:00.539040 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9g22g" Jul 17 18:19:00.541756 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:00.541736 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:19:00.541885 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:00.541768 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dpsm\"" Jul 17 18:19:00.546021 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:00.545993 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9g22g"] Jul 17 18:19:00.697912 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:00.697869 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be3cf669-64c9-4818-a8eb-b05a1a63297d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-9g22g\" (UID: \"be3cf669-64c9-4818-a8eb-b05a1a63297d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9g22g" Jul 17 18:19:00.799350 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:00.799247 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be3cf669-64c9-4818-a8eb-b05a1a63297d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-9g22g\" (UID: \"be3cf669-64c9-4818-a8eb-b05a1a63297d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9g22g" Jul 17 18:19:00.799894 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:00.799872 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be3cf669-64c9-4818-a8eb-b05a1a63297d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-9g22g\" (UID: \"be3cf669-64c9-4818-a8eb-b05a1a63297d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9g22g" Jul 17 18:19:00.851469 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:00.851436 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9g22g" Jul 17 18:19:00.987626 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:00.987606 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9g22g"] Jul 17 18:19:00.990120 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:19:00.990093 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3cf669_64c9_4818_a8eb_b05a1a63297d.slice/crio-2936cf8b3adb17c4a67ed61d81202bc3d7f8a25a3dfb2878ddfbb5e84db06bf4 WatchSource:0}: Error finding container 2936cf8b3adb17c4a67ed61d81202bc3d7f8a25a3dfb2878ddfbb5e84db06bf4: Status 404 returned error can't find the container with id 2936cf8b3adb17c4a67ed61d81202bc3d7f8a25a3dfb2878ddfbb5e84db06bf4 Jul 17 18:19:01.092445 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:01.092344 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9g22g" event={"ID":"be3cf669-64c9-4818-a8eb-b05a1a63297d","Type":"ContainerStarted","Data":"2936cf8b3adb17c4a67ed61d81202bc3d7f8a25a3dfb2878ddfbb5e84db06bf4"} Jul 17 18:19:02.098713 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:02.098674 2574 generic.go:356] "Generic (PLEG): container finished" podID="be3cf669-64c9-4818-a8eb-b05a1a63297d" containerID="87ca5532fe39f927fe400ce5cf84db478f66c6f43e1c831a71571a289d9823f4" exitCode=7 Jul 17 18:19:02.099073 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:02.098769 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9g22g" event={"ID":"be3cf669-64c9-4818-a8eb-b05a1a63297d","Type":"ContainerDied","Data":"87ca5532fe39f927fe400ce5cf84db478f66c6f43e1c831a71571a289d9823f4"} Jul 17 18:19:02.099073 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:02.098944 2574 scope.go:117] "RemoveContainer" containerID="87ca5532fe39f927fe400ce5cf84db478f66c6f43e1c831a71571a289d9823f4" Jul 17 18:19:03.105829 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:03.105728 2574 generic.go:356] "Generic (PLEG): container finished" podID="be3cf669-64c9-4818-a8eb-b05a1a63297d" containerID="bff5d47b1e02ca51d4c7e700bae467e3eaa80e364f3d125f30cbe4c8dec2f181" exitCode=7 Jul 17 18:19:03.105829 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:03.105769 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9g22g" event={"ID":"be3cf669-64c9-4818-a8eb-b05a1a63297d","Type":"ContainerDied","Data":"bff5d47b1e02ca51d4c7e700bae467e3eaa80e364f3d125f30cbe4c8dec2f181"} Jul 17 18:19:03.105829 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:03.105818 2574 scope.go:117] "RemoveContainer" containerID="87ca5532fe39f927fe400ce5cf84db478f66c6f43e1c831a71571a289d9823f4" Jul 17 18:19:03.106405 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:03.106075 2574 scope.go:117] "RemoveContainer" containerID="bff5d47b1e02ca51d4c7e700bae467e3eaa80e364f3d125f30cbe4c8dec2f181" Jul 17 18:19:03.106405 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:19:03.106324 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-9g22g_odh-ai-gateway-infra(be3cf669-64c9-4818-a8eb-b05a1a63297d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9g22g" podUID="be3cf669-64c9-4818-a8eb-b05a1a63297d" Jul 17 18:19:04.112458 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:04.112432 2574 scope.go:117] "RemoveContainer" containerID="bff5d47b1e02ca51d4c7e700bae467e3eaa80e364f3d125f30cbe4c8dec2f181" Jul 17 18:19:04.112837 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:19:04.112651 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-9g22g_odh-ai-gateway-infra(be3cf669-64c9-4818-a8eb-b05a1a63297d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9g22g" podUID="be3cf669-64c9-4818-a8eb-b05a1a63297d" Jul 17 18:19:10.654447 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:10.654410 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:10.753860 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:10.753823 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:16.432167 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:16.432119 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:16.658531 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:16.658493 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:16.980603 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:16.980572 2574 scope.go:117] "RemoveContainer" containerID="bff5d47b1e02ca51d4c7e700bae467e3eaa80e364f3d125f30cbe4c8dec2f181" Jul 17 18:19:17.137547 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:17.137509 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:18.185239 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:18.185207 2574 generic.go:356] "Generic (PLEG): container finished" podID="be3cf669-64c9-4818-a8eb-b05a1a63297d" containerID="4dfa12434c0dde07ff0216f263223986aa91429e28a3cfdc1e7a571d98faa3d2" exitCode=6 Jul 17 18:19:18.185704 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:18.185278 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9g22g" event={"ID":"be3cf669-64c9-4818-a8eb-b05a1a63297d","Type":"ContainerDied","Data":"4dfa12434c0dde07ff0216f263223986aa91429e28a3cfdc1e7a571d98faa3d2"} Jul 17 18:19:18.185704 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:18.185341 2574 scope.go:117] "RemoveContainer" containerID="bff5d47b1e02ca51d4c7e700bae467e3eaa80e364f3d125f30cbe4c8dec2f181" Jul 17 18:19:18.185704 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:18.185651 2574 scope.go:117] "RemoveContainer" containerID="4dfa12434c0dde07ff0216f263223986aa91429e28a3cfdc1e7a571d98faa3d2" Jul 17 18:19:18.185938 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:19:18.185920 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-9g22g_odh-ai-gateway-infra(be3cf669-64c9-4818-a8eb-b05a1a63297d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9g22g" podUID="be3cf669-64c9-4818-a8eb-b05a1a63297d" Jul 17 18:19:19.209438 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:19.209407 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9g22g"] Jul 17 18:19:19.350449 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:19.350418 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9g22g" Jul 17 18:19:19.369748 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:19.369715 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be3cf669-64c9-4818-a8eb-b05a1a63297d-maas-api-service-ca\") pod \"be3cf669-64c9-4818-a8eb-b05a1a63297d\" (UID: \"be3cf669-64c9-4818-a8eb-b05a1a63297d\") " Jul 17 18:19:19.370094 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:19.370069 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be3cf669-64c9-4818-a8eb-b05a1a63297d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "be3cf669-64c9-4818-a8eb-b05a1a63297d" (UID: "be3cf669-64c9-4818-a8eb-b05a1a63297d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:19:19.470536 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:19.470488 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be3cf669-64c9-4818-a8eb-b05a1a63297d-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:19:20.199172 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:20.199133 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9g22g" event={"ID":"be3cf669-64c9-4818-a8eb-b05a1a63297d","Type":"ContainerDied","Data":"2936cf8b3adb17c4a67ed61d81202bc3d7f8a25a3dfb2878ddfbb5e84db06bf4"} Jul 17 18:19:20.199365 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:20.199190 2574 scope.go:117] "RemoveContainer" containerID="4dfa12434c0dde07ff0216f263223986aa91429e28a3cfdc1e7a571d98faa3d2" Jul 17 18:19:20.199365 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:20.199142 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9g22g" Jul 17 18:19:20.222431 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:20.222398 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9g22g"] Jul 17 18:19:20.226130 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:20.226105 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9g22g"] Jul 17 18:19:20.986798 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:20.986766 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be3cf669-64c9-4818-a8eb-b05a1a63297d" path="/var/lib/kubelet/pods/be3cf669-64c9-4818-a8eb-b05a1a63297d/volumes" Jul 17 18:19:22.374463 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:22.374425 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:22.487002 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:22.486967 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:22.685489 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:22.685452 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:22.772321 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:22.772288 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j"] Jul 17 18:19:22.772722 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:22.772710 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be3cf669-64c9-4818-a8eb-b05a1a63297d" containerName="revoke-keys" Jul 17 18:19:22.772771 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:22.772725 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="be3cf669-64c9-4818-a8eb-b05a1a63297d" containerName="revoke-keys" Jul 17 18:19:22.772771 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:22.772746 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be3cf669-64c9-4818-a8eb-b05a1a63297d" containerName="revoke-keys" Jul 17 18:19:22.772771 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:22.772751 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="be3cf669-64c9-4818-a8eb-b05a1a63297d" containerName="revoke-keys" Jul 17 18:19:22.772873 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:22.772804 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="be3cf669-64c9-4818-a8eb-b05a1a63297d" containerName="revoke-keys" Jul 17 18:19:22.772873 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:22.772814 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a6cf51b-5947-40dd-8afa-92e5943d4e2e" containerName="revoke-keys" Jul 17 18:19:22.772873 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:22.772820 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="be3cf669-64c9-4818-a8eb-b05a1a63297d" containerName="revoke-keys" Jul 17 18:19:22.777403 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:22.777382 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j" Jul 17 18:19:22.779894 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:22.779862 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:19:22.779894 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:22.779882 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dpsm\"" Jul 17 18:19:22.784391 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:22.784346 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j"] Jul 17 18:19:22.786020 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:22.785996 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:22.801658 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:22.801623 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03fc188b-db8d-46ef-a59b-f5b722d491a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j\" (UID: \"03fc188b-db8d-46ef-a59b-f5b722d491a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j" Jul 17 18:19:22.903110 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:22.903021 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03fc188b-db8d-46ef-a59b-f5b722d491a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j\" (UID: \"03fc188b-db8d-46ef-a59b-f5b722d491a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j" Jul 17 18:19:22.903755 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:22.903727 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03fc188b-db8d-46ef-a59b-f5b722d491a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j\" (UID: \"03fc188b-db8d-46ef-a59b-f5b722d491a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j" Jul 17 18:19:23.090746 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:23.090692 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j" Jul 17 18:19:23.229488 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:23.229460 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j"] Jul 17 18:19:23.231910 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:19:23.231873 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03fc188b_db8d_46ef_a59b_f5b722d491a4.slice/crio-e57acc8100928b049ac9be949d25a75a579c70eb9213a734dc9cf32c328f898d WatchSource:0}: Error finding container e57acc8100928b049ac9be949d25a75a579c70eb9213a734dc9cf32c328f898d: Status 404 returned error can't find the container with id e57acc8100928b049ac9be949d25a75a579c70eb9213a734dc9cf32c328f898d Jul 17 18:19:23.814338 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:23.814235 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:23.910435 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:23.910404 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:24.015017 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:24.014992 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:24.121562 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:24.121450 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:24.230056 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:24.230018 2574 generic.go:356] "Generic (PLEG): container finished" podID="03fc188b-db8d-46ef-a59b-f5b722d491a4" containerID="30fcf27580701bf99832ddb00da76b4d18f33c47bcec408fbe4b0b0772f02965" exitCode=6 Jul 17 18:19:24.230056 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:24.230063 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j" event={"ID":"03fc188b-db8d-46ef-a59b-f5b722d491a4","Type":"ContainerDied","Data":"30fcf27580701bf99832ddb00da76b4d18f33c47bcec408fbe4b0b0772f02965"} Jul 17 18:19:24.230339 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:24.230102 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j" event={"ID":"03fc188b-db8d-46ef-a59b-f5b722d491a4","Type":"ContainerStarted","Data":"e57acc8100928b049ac9be949d25a75a579c70eb9213a734dc9cf32c328f898d"} Jul 17 18:19:24.230339 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:24.230313 2574 scope.go:117] "RemoveContainer" containerID="30fcf27580701bf99832ddb00da76b4d18f33c47bcec408fbe4b0b0772f02965" Jul 17 18:19:24.450651 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:24.450612 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:24.549156 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:24.549118 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:25.237298 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:25.237267 2574 generic.go:356] "Generic (PLEG): container finished" podID="03fc188b-db8d-46ef-a59b-f5b722d491a4" containerID="3e9212a60b7df0127a97c38e2d1f77d6d3a422049872b8f688bd6d1ab4a7977e" exitCode=6 Jul 17 18:19:25.237805 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:25.237315 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j" event={"ID":"03fc188b-db8d-46ef-a59b-f5b722d491a4","Type":"ContainerDied","Data":"3e9212a60b7df0127a97c38e2d1f77d6d3a422049872b8f688bd6d1ab4a7977e"} Jul 17 18:19:25.237805 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:25.237376 2574 scope.go:117] "RemoveContainer" containerID="30fcf27580701bf99832ddb00da76b4d18f33c47bcec408fbe4b0b0772f02965" Jul 17 18:19:25.237805 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:25.237652 2574 scope.go:117] "RemoveContainer" containerID="3e9212a60b7df0127a97c38e2d1f77d6d3a422049872b8f688bd6d1ab4a7977e" Jul 17 18:19:25.238000 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:19:25.237904 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j_odh-ai-gateway-infra(03fc188b-db8d-46ef-a59b-f5b722d491a4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j" podUID="03fc188b-db8d-46ef-a59b-f5b722d491a4" Jul 17 18:19:26.091158 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:26.091114 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:19:26.246259 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:26.246235 2574 scope.go:117] "RemoveContainer" containerID="3e9212a60b7df0127a97c38e2d1f77d6d3a422049872b8f688bd6d1ab4a7977e" Jul 17 18:19:26.246675 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:19:26.246445 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j_odh-ai-gateway-infra(03fc188b-db8d-46ef-a59b-f5b722d491a4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j" podUID="03fc188b-db8d-46ef-a59b-f5b722d491a4" Jul 17 18:19:28.991855 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:28.991826 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq"] Jul 17 18:19:28.992258 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:28.992243 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be3cf669-64c9-4818-a8eb-b05a1a63297d" containerName="revoke-keys" Jul 17 18:19:28.992303 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:28.992260 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="be3cf669-64c9-4818-a8eb-b05a1a63297d" containerName="revoke-keys" Jul 17 18:19:28.992368 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:28.992344 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="be3cf669-64c9-4818-a8eb-b05a1a63297d" containerName="revoke-keys" Jul 17 18:19:28.996738 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:28.996719 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq" Jul 17 18:19:29.001241 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:29.001218 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq"] Jul 17 18:19:29.061330 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:29.061287 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b51517-260b-45d3-9c5c-362d4eb3ed1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq\" (UID: \"f2b51517-260b-45d3-9c5c-362d4eb3ed1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq" Jul 17 18:19:29.162285 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:29.162249 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b51517-260b-45d3-9c5c-362d4eb3ed1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq\" (UID: \"f2b51517-260b-45d3-9c5c-362d4eb3ed1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq" Jul 17 18:19:29.162877 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:29.162855 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b51517-260b-45d3-9c5c-362d4eb3ed1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq\" (UID: \"f2b51517-260b-45d3-9c5c-362d4eb3ed1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq" Jul 17 18:19:29.310995 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:29.310910 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq" Jul 17 18:19:29.454189 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:29.454166 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq"] Jul 17 18:19:29.456148 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:19:29.456118 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b51517_260b_45d3_9c5c_362d4eb3ed1c.slice/crio-0088556da3186fb896e5682531c2dec7fd3afc03b4161707ffe853063e5b954c WatchSource:0}: Error finding container 0088556da3186fb896e5682531c2dec7fd3afc03b4161707ffe853063e5b954c: Status 404 returned error can't find the container with id 0088556da3186fb896e5682531c2dec7fd3afc03b4161707ffe853063e5b954c Jul 17 18:19:30.268750 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:30.268712 2574 generic.go:356] "Generic (PLEG): container finished" podID="f2b51517-260b-45d3-9c5c-362d4eb3ed1c" containerID="d45bf229a553f5188fc8b2ef85cda59e2a1c9d90e223ed294a5825eb2e460e3c" exitCode=7 Jul 17 18:19:30.269145 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:30.268758 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq" event={"ID":"f2b51517-260b-45d3-9c5c-362d4eb3ed1c","Type":"ContainerDied","Data":"d45bf229a553f5188fc8b2ef85cda59e2a1c9d90e223ed294a5825eb2e460e3c"} Jul 17 18:19:30.269145 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:30.268806 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq" event={"ID":"f2b51517-260b-45d3-9c5c-362d4eb3ed1c","Type":"ContainerStarted","Data":"0088556da3186fb896e5682531c2dec7fd3afc03b4161707ffe853063e5b954c"} Jul 17 18:19:30.269145 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:30.268986 2574 scope.go:117] "RemoveContainer" containerID="d45bf229a553f5188fc8b2ef85cda59e2a1c9d90e223ed294a5825eb2e460e3c" Jul 17 18:19:31.276004 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:31.275917 2574 generic.go:356] "Generic (PLEG): container finished" podID="f2b51517-260b-45d3-9c5c-362d4eb3ed1c" containerID="6989a9b69aa817ff261d1b785912addd2b25d1f8df9760551bb5be40c409c3ef" exitCode=7 Jul 17 18:19:31.276441 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:31.276011 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq" event={"ID":"f2b51517-260b-45d3-9c5c-362d4eb3ed1c","Type":"ContainerDied","Data":"6989a9b69aa817ff261d1b785912addd2b25d1f8df9760551bb5be40c409c3ef"} Jul 17 18:19:31.276441 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:31.276073 2574 scope.go:117] "RemoveContainer" containerID="d45bf229a553f5188fc8b2ef85cda59e2a1c9d90e223ed294a5825eb2e460e3c" Jul 17 18:19:31.276441 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:31.276309 2574 scope.go:117] "RemoveContainer" containerID="6989a9b69aa817ff261d1b785912addd2b25d1f8df9760551bb5be40c409c3ef" Jul 17 18:19:31.276608 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:19:31.276588 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq_odh-ai-gateway-infra(f2b51517-260b-45d3-9c5c-362d4eb3ed1c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq" podUID="f2b51517-260b-45d3-9c5c-362d4eb3ed1c" Jul 17 18:19:32.283993 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:32.283960 2574 scope.go:117] "RemoveContainer" containerID="6989a9b69aa817ff261d1b785912addd2b25d1f8df9760551bb5be40c409c3ef" Jul 17 18:19:32.284498 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:19:32.284227 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq_odh-ai-gateway-infra(f2b51517-260b-45d3-9c5c-362d4eb3ed1c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq" podUID="f2b51517-260b-45d3-9c5c-362d4eb3ed1c" Jul 17 18:19:37.980266 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:37.980228 2574 scope.go:117] "RemoveContainer" containerID="3e9212a60b7df0127a97c38e2d1f77d6d3a422049872b8f688bd6d1ab4a7977e" Jul 17 18:19:38.320816 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:38.320774 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j" event={"ID":"03fc188b-db8d-46ef-a59b-f5b722d491a4","Type":"ContainerStarted","Data":"97be35966ee712d1a65a1510a844340f2067db866397b672f2c1395d8d0b4243"} Jul 17 18:19:38.337266 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:38.337187 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j" podStartSLOduration=16.337166184 podStartE2EDuration="16.337166184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:19:38.334993638 +0000 UTC m=+1789.922946330" watchObservedRunningTime="2026-07-17 18:19:38.337166184 +0000 UTC m=+1789.925118875" Jul 17 18:19:39.327878 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:39.327846 2574 generic.go:356] "Generic (PLEG): container finished" podID="03fc188b-db8d-46ef-a59b-f5b722d491a4" containerID="97be35966ee712d1a65a1510a844340f2067db866397b672f2c1395d8d0b4243" exitCode=6 Jul 17 18:19:39.328312 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:39.327909 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j" event={"ID":"03fc188b-db8d-46ef-a59b-f5b722d491a4","Type":"ContainerDied","Data":"97be35966ee712d1a65a1510a844340f2067db866397b672f2c1395d8d0b4243"} Jul 17 18:19:39.328312 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:39.327961 2574 scope.go:117] "RemoveContainer" containerID="3e9212a60b7df0127a97c38e2d1f77d6d3a422049872b8f688bd6d1ab4a7977e" Jul 17 18:19:39.328312 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:39.328254 2574 scope.go:117] "RemoveContainer" containerID="97be35966ee712d1a65a1510a844340f2067db866397b672f2c1395d8d0b4243" Jul 17 18:19:39.328547 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:19:39.328521 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j_odh-ai-gateway-infra(03fc188b-db8d-46ef-a59b-f5b722d491a4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j" podUID="03fc188b-db8d-46ef-a59b-f5b722d491a4" Jul 17 18:19:39.351262 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:39.351230 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j"] Jul 17 18:19:40.487779 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:40.487751 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j" Jul 17 18:19:40.567970 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:40.567922 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03fc188b-db8d-46ef-a59b-f5b722d491a4-maas-api-service-ca\") pod \"03fc188b-db8d-46ef-a59b-f5b722d491a4\" (UID: \"03fc188b-db8d-46ef-a59b-f5b722d491a4\") " Jul 17 18:19:40.568278 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:40.568260 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03fc188b-db8d-46ef-a59b-f5b722d491a4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "03fc188b-db8d-46ef-a59b-f5b722d491a4" (UID: "03fc188b-db8d-46ef-a59b-f5b722d491a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:19:40.669925 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:40.669826 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03fc188b-db8d-46ef-a59b-f5b722d491a4-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:19:41.341209 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:41.341129 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j" Jul 17 18:19:41.341209 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:41.341175 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j" event={"ID":"03fc188b-db8d-46ef-a59b-f5b722d491a4","Type":"ContainerDied","Data":"e57acc8100928b049ac9be949d25a75a579c70eb9213a734dc9cf32c328f898d"} Jul 17 18:19:41.341457 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:41.341240 2574 scope.go:117] "RemoveContainer" containerID="97be35966ee712d1a65a1510a844340f2067db866397b672f2c1395d8d0b4243" Jul 17 18:19:41.357307 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:41.357278 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j"] Jul 17 18:19:41.361327 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:41.361300 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-clr8j"] Jul 17 18:19:42.985733 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:42.985699 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03fc188b-db8d-46ef-a59b-f5b722d491a4" path="/var/lib/kubelet/pods/03fc188b-db8d-46ef-a59b-f5b722d491a4/volumes" Jul 17 18:19:45.979417 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:45.979382 2574 scope.go:117] "RemoveContainer" containerID="6989a9b69aa817ff261d1b785912addd2b25d1f8df9760551bb5be40c409c3ef" Jul 17 18:19:46.368265 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:46.368178 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq" event={"ID":"f2b51517-260b-45d3-9c5c-362d4eb3ed1c","Type":"ContainerStarted","Data":"a0eb4a83c682965daddd8f67b926cee49a71185574fc1a0ab96ee56bc405ea01"} Jul 17 18:19:46.383450 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:46.383395 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq" podStartSLOduration=18.383380644 podStartE2EDuration="18.383380644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:19:46.382385774 +0000 UTC m=+1797.970338462" watchObservedRunningTime="2026-07-17 18:19:46.383380644 +0000 UTC m=+1797.971333334" Jul 17 18:19:47.375219 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:47.375127 2574 generic.go:356] "Generic (PLEG): container finished" podID="f2b51517-260b-45d3-9c5c-362d4eb3ed1c" containerID="a0eb4a83c682965daddd8f67b926cee49a71185574fc1a0ab96ee56bc405ea01" exitCode=7 Jul 17 18:19:47.375219 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:47.375198 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq" event={"ID":"f2b51517-260b-45d3-9c5c-362d4eb3ed1c","Type":"ContainerDied","Data":"a0eb4a83c682965daddd8f67b926cee49a71185574fc1a0ab96ee56bc405ea01"} Jul 17 18:19:47.375711 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:47.375274 2574 scope.go:117] "RemoveContainer" containerID="6989a9b69aa817ff261d1b785912addd2b25d1f8df9760551bb5be40c409c3ef" Jul 17 18:19:47.375711 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:47.375549 2574 scope.go:117] "RemoveContainer" containerID="a0eb4a83c682965daddd8f67b926cee49a71185574fc1a0ab96ee56bc405ea01" Jul 17 18:19:47.375823 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:19:47.375805 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq_odh-ai-gateway-infra(f2b51517-260b-45d3-9c5c-362d4eb3ed1c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq" podUID="f2b51517-260b-45d3-9c5c-362d4eb3ed1c" Jul 17 18:19:47.399288 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:47.399259 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq"] Jul 17 18:19:48.520161 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:48.520136 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq" Jul 17 18:19:48.646749 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:48.646658 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b51517-260b-45d3-9c5c-362d4eb3ed1c-maas-api-service-ca\") pod \"f2b51517-260b-45d3-9c5c-362d4eb3ed1c\" (UID: \"f2b51517-260b-45d3-9c5c-362d4eb3ed1c\") " Jul 17 18:19:48.647034 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:48.647008 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b51517-260b-45d3-9c5c-362d4eb3ed1c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f2b51517-260b-45d3-9c5c-362d4eb3ed1c" (UID: "f2b51517-260b-45d3-9c5c-362d4eb3ed1c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:19:48.748172 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:48.748128 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2b51517-260b-45d3-9c5c-362d4eb3ed1c-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:19:49.392427 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:49.392328 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq" Jul 17 18:19:49.392427 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:49.392378 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq" event={"ID":"f2b51517-260b-45d3-9c5c-362d4eb3ed1c","Type":"ContainerDied","Data":"0088556da3186fb896e5682531c2dec7fd3afc03b4161707ffe853063e5b954c"} Jul 17 18:19:49.392634 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:49.392444 2574 scope.go:117] "RemoveContainer" containerID="a0eb4a83c682965daddd8f67b926cee49a71185574fc1a0ab96ee56bc405ea01" Jul 17 18:19:49.410164 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:49.410133 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq"] Jul 17 18:19:49.417453 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:49.417428 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-9gzgq"] Jul 17 18:19:50.573240 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.573206 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4"] Jul 17 18:19:50.573717 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.573624 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03fc188b-db8d-46ef-a59b-f5b722d491a4" containerName="revoke-keys" Jul 17 18:19:50.573717 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.573636 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="03fc188b-db8d-46ef-a59b-f5b722d491a4" containerName="revoke-keys" Jul 17 18:19:50.573717 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.573654 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b51517-260b-45d3-9c5c-362d4eb3ed1c" containerName="revoke-keys" Jul 17 18:19:50.573717 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.573660 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b51517-260b-45d3-9c5c-362d4eb3ed1c" containerName="revoke-keys" Jul 17 18:19:50.573717 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.573666 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03fc188b-db8d-46ef-a59b-f5b722d491a4" containerName="revoke-keys" Jul 17 18:19:50.573717 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.573671 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="03fc188b-db8d-46ef-a59b-f5b722d491a4" containerName="revoke-keys" Jul 17 18:19:50.573717 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.573678 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b51517-260b-45d3-9c5c-362d4eb3ed1c" containerName="revoke-keys" Jul 17 18:19:50.573717 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.573684 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b51517-260b-45d3-9c5c-362d4eb3ed1c" containerName="revoke-keys" Jul 17 18:19:50.573717 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.573692 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03fc188b-db8d-46ef-a59b-f5b722d491a4" containerName="revoke-keys" Jul 17 18:19:50.573717 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.573697 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="03fc188b-db8d-46ef-a59b-f5b722d491a4" containerName="revoke-keys" Jul 17 18:19:50.574023 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.573774 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2b51517-260b-45d3-9c5c-362d4eb3ed1c" containerName="revoke-keys" Jul 17 18:19:50.574023 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.573784 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="03fc188b-db8d-46ef-a59b-f5b722d491a4" containerName="revoke-keys" Jul 17 18:19:50.574023 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.573792 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="03fc188b-db8d-46ef-a59b-f5b722d491a4" containerName="revoke-keys" Jul 17 18:19:50.574023 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.573798 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2b51517-260b-45d3-9c5c-362d4eb3ed1c" containerName="revoke-keys" Jul 17 18:19:50.574023 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.573804 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="03fc188b-db8d-46ef-a59b-f5b722d491a4" containerName="revoke-keys" Jul 17 18:19:50.576271 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.576254 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4" Jul 17 18:19:50.578649 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.578622 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dpsm\"" Jul 17 18:19:50.578649 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.578624 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:19:50.582942 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.582916 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4"] Jul 17 18:19:50.664350 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.664312 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf9541e9-83b5-4e76-8319-1d58a8fc8c57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4\" (UID: \"cf9541e9-83b5-4e76-8319-1d58a8fc8c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4" Jul 17 18:19:50.765738 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.765704 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf9541e9-83b5-4e76-8319-1d58a8fc8c57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4\" (UID: \"cf9541e9-83b5-4e76-8319-1d58a8fc8c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4" Jul 17 18:19:50.766327 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.766299 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf9541e9-83b5-4e76-8319-1d58a8fc8c57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4\" (UID: \"cf9541e9-83b5-4e76-8319-1d58a8fc8c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4" Jul 17 18:19:50.889619 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.889531 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4" Jul 17 18:19:50.987986 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:50.987950 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2b51517-260b-45d3-9c5c-362d4eb3ed1c" path="/var/lib/kubelet/pods/f2b51517-260b-45d3-9c5c-362d4eb3ed1c/volumes" Jul 17 18:19:51.028577 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:51.028551 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4"] Jul 17 18:19:51.031034 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:19:51.031003 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9541e9_83b5_4e76_8319_1d58a8fc8c57.slice/crio-0892c4a1c0a02f37abdb841fa40e62db85a262062da36651833fda74ba04ddaf WatchSource:0}: Error finding container 0892c4a1c0a02f37abdb841fa40e62db85a262062da36651833fda74ba04ddaf: Status 404 returned error can't find the container with id 0892c4a1c0a02f37abdb841fa40e62db85a262062da36651833fda74ba04ddaf Jul 17 18:19:51.405404 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:51.405332 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4" event={"ID":"cf9541e9-83b5-4e76-8319-1d58a8fc8c57","Type":"ContainerStarted","Data":"dd1a55f49c48b75b24474ef04439599cab33ef13c12130186201d387c74c48ae"} Jul 17 18:19:51.405588 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:51.405421 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4" event={"ID":"cf9541e9-83b5-4e76-8319-1d58a8fc8c57","Type":"ContainerStarted","Data":"0892c4a1c0a02f37abdb841fa40e62db85a262062da36651833fda74ba04ddaf"} Jul 17 18:19:51.421776 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:51.421722 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4" podStartSLOduration=1.421706276 podStartE2EDuration="1.421706276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:19:51.419531207 +0000 UTC m=+1803.007483897" watchObservedRunningTime="2026-07-17 18:19:51.421706276 +0000 UTC m=+1803.009658966" Jul 17 18:19:52.412247 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:52.412213 2574 generic.go:356] "Generic (PLEG): container finished" podID="cf9541e9-83b5-4e76-8319-1d58a8fc8c57" containerID="dd1a55f49c48b75b24474ef04439599cab33ef13c12130186201d387c74c48ae" exitCode=6 Jul 17 18:19:52.412763 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:52.412276 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4" event={"ID":"cf9541e9-83b5-4e76-8319-1d58a8fc8c57","Type":"ContainerDied","Data":"dd1a55f49c48b75b24474ef04439599cab33ef13c12130186201d387c74c48ae"} Jul 17 18:19:52.412763 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:52.412572 2574 scope.go:117] "RemoveContainer" containerID="dd1a55f49c48b75b24474ef04439599cab33ef13c12130186201d387c74c48ae" Jul 17 18:19:53.418791 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:53.418690 2574 generic.go:356] "Generic (PLEG): container finished" podID="cf9541e9-83b5-4e76-8319-1d58a8fc8c57" containerID="9ee538376e7462804cb07901f95bac0cda795793f260873bb58fce565e291325" exitCode=6 Jul 17 18:19:53.418791 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:53.418755 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4" event={"ID":"cf9541e9-83b5-4e76-8319-1d58a8fc8c57","Type":"ContainerDied","Data":"9ee538376e7462804cb07901f95bac0cda795793f260873bb58fce565e291325"} Jul 17 18:19:53.419278 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:53.418797 2574 scope.go:117] "RemoveContainer" containerID="dd1a55f49c48b75b24474ef04439599cab33ef13c12130186201d387c74c48ae" Jul 17 18:19:53.419278 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:53.418970 2574 scope.go:117] "RemoveContainer" containerID="9ee538376e7462804cb07901f95bac0cda795793f260873bb58fce565e291325" Jul 17 18:19:53.419278 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:19:53.419222 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4_odh-ai-gateway-infra(cf9541e9-83b5-4e76-8319-1d58a8fc8c57)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4" podUID="cf9541e9-83b5-4e76-8319-1d58a8fc8c57" Jul 17 18:19:54.426082 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:19:54.426055 2574 scope.go:117] "RemoveContainer" containerID="9ee538376e7462804cb07901f95bac0cda795793f260873bb58fce565e291325" Jul 17 18:19:54.426497 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:19:54.426296 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4_odh-ai-gateway-infra(cf9541e9-83b5-4e76-8319-1d58a8fc8c57)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4" podUID="cf9541e9-83b5-4e76-8319-1d58a8fc8c57" Jul 17 18:20:07.980315 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:07.980275 2574 scope.go:117] "RemoveContainer" containerID="9ee538376e7462804cb07901f95bac0cda795793f260873bb58fce565e291325" Jul 17 18:20:08.503366 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:08.503330 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4" event={"ID":"cf9541e9-83b5-4e76-8319-1d58a8fc8c57","Type":"ContainerStarted","Data":"19dd716b6ac2040f6b638007a49783602b31af8e381a5a4af48fb917282741b1"} Jul 17 18:20:09.510572 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:09.510533 2574 generic.go:356] "Generic (PLEG): container finished" podID="cf9541e9-83b5-4e76-8319-1d58a8fc8c57" containerID="19dd716b6ac2040f6b638007a49783602b31af8e381a5a4af48fb917282741b1" exitCode=6 Jul 17 18:20:09.511013 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:09.510609 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4" event={"ID":"cf9541e9-83b5-4e76-8319-1d58a8fc8c57","Type":"ContainerDied","Data":"19dd716b6ac2040f6b638007a49783602b31af8e381a5a4af48fb917282741b1"} Jul 17 18:20:09.511013 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:09.510675 2574 scope.go:117] "RemoveContainer" containerID="9ee538376e7462804cb07901f95bac0cda795793f260873bb58fce565e291325" Jul 17 18:20:09.511013 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:09.510977 2574 scope.go:117] "RemoveContainer" containerID="19dd716b6ac2040f6b638007a49783602b31af8e381a5a4af48fb917282741b1" Jul 17 18:20:09.511255 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:20:09.511220 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4_odh-ai-gateway-infra(cf9541e9-83b5-4e76-8319-1d58a8fc8c57)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4" podUID="cf9541e9-83b5-4e76-8319-1d58a8fc8c57" Jul 17 18:20:09.535684 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:09.535646 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4"] Jul 17 18:20:10.652562 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:10.652535 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4" Jul 17 18:20:10.763840 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:10.763802 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf9541e9-83b5-4e76-8319-1d58a8fc8c57-maas-api-service-ca\") pod \"cf9541e9-83b5-4e76-8319-1d58a8fc8c57\" (UID: \"cf9541e9-83b5-4e76-8319-1d58a8fc8c57\") " Jul 17 18:20:10.764203 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:10.764182 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9541e9-83b5-4e76-8319-1d58a8fc8c57-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cf9541e9-83b5-4e76-8319-1d58a8fc8c57" (UID: "cf9541e9-83b5-4e76-8319-1d58a8fc8c57"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:20:10.865037 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:10.864945 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf9541e9-83b5-4e76-8319-1d58a8fc8c57-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:20:11.523219 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:11.523187 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4" Jul 17 18:20:11.523447 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:11.523220 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4" event={"ID":"cf9541e9-83b5-4e76-8319-1d58a8fc8c57","Type":"ContainerDied","Data":"0892c4a1c0a02f37abdb841fa40e62db85a262062da36651833fda74ba04ddaf"} Jul 17 18:20:11.523447 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:11.523281 2574 scope.go:117] "RemoveContainer" containerID="19dd716b6ac2040f6b638007a49783602b31af8e381a5a4af48fb917282741b1" Jul 17 18:20:11.545381 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:11.545333 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4"] Jul 17 18:20:11.550846 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:11.550822 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-8nzk4"] Jul 17 18:20:12.800440 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:12.800403 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8"] Jul 17 18:20:12.801046 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:12.801025 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf9541e9-83b5-4e76-8319-1d58a8fc8c57" containerName="revoke-keys" Jul 17 18:20:12.801141 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:12.801050 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf9541e9-83b5-4e76-8319-1d58a8fc8c57" containerName="revoke-keys" Jul 17 18:20:12.801141 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:12.801065 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2b51517-260b-45d3-9c5c-362d4eb3ed1c" containerName="revoke-keys" Jul 17 18:20:12.801141 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:12.801073 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2b51517-260b-45d3-9c5c-362d4eb3ed1c" containerName="revoke-keys" Jul 17 18:20:12.801141 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:12.801082 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf9541e9-83b5-4e76-8319-1d58a8fc8c57" containerName="revoke-keys" Jul 17 18:20:12.801141 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:12.801090 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf9541e9-83b5-4e76-8319-1d58a8fc8c57" containerName="revoke-keys" Jul 17 18:20:12.801141 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:12.801110 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf9541e9-83b5-4e76-8319-1d58a8fc8c57" containerName="revoke-keys" Jul 17 18:20:12.801141 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:12.801119 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf9541e9-83b5-4e76-8319-1d58a8fc8c57" containerName="revoke-keys" Jul 17 18:20:12.801520 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:12.801197 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf9541e9-83b5-4e76-8319-1d58a8fc8c57" containerName="revoke-keys" Jul 17 18:20:12.801520 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:12.801210 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf9541e9-83b5-4e76-8319-1d58a8fc8c57" containerName="revoke-keys" Jul 17 18:20:12.801520 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:12.801220 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf9541e9-83b5-4e76-8319-1d58a8fc8c57" containerName="revoke-keys" Jul 17 18:20:12.801520 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:12.801232 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2b51517-260b-45d3-9c5c-362d4eb3ed1c" containerName="revoke-keys" Jul 17 18:20:12.805686 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:12.805664 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8" Jul 17 18:20:12.808668 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:12.808644 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dpsm\"" Jul 17 18:20:12.808876 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:12.808684 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:20:12.810820 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:12.810777 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8"] Jul 17 18:20:12.885608 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:12.885567 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d530976f-894a-46f3-867d-3a41f32abd96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8\" (UID: \"d530976f-894a-46f3-867d-3a41f32abd96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8" Jul 17 18:20:12.985704 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:12.985671 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf9541e9-83b5-4e76-8319-1d58a8fc8c57" path="/var/lib/kubelet/pods/cf9541e9-83b5-4e76-8319-1d58a8fc8c57/volumes" Jul 17 18:20:12.986284 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:12.986268 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d530976f-894a-46f3-867d-3a41f32abd96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8\" (UID: \"d530976f-894a-46f3-867d-3a41f32abd96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8" Jul 17 18:20:12.986872 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:12.986853 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d530976f-894a-46f3-867d-3a41f32abd96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8\" (UID: \"d530976f-894a-46f3-867d-3a41f32abd96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8" Jul 17 18:20:13.118171 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:13.118077 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8" Jul 17 18:20:13.253640 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:13.253615 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8"] Jul 17 18:20:13.255964 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:20:13.255941 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd530976f_894a_46f3_867d_3a41f32abd96.slice/crio-91c4d177359d0771a940cb54140a68ff997944e8d9f72a1c89f2ecb055a6cb6e WatchSource:0}: Error finding container 91c4d177359d0771a940cb54140a68ff997944e8d9f72a1c89f2ecb055a6cb6e: Status 404 returned error can't find the container with id 91c4d177359d0771a940cb54140a68ff997944e8d9f72a1c89f2ecb055a6cb6e Jul 17 18:20:13.535528 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:13.535493 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8" event={"ID":"d530976f-894a-46f3-867d-3a41f32abd96","Type":"ContainerStarted","Data":"91c4d177359d0771a940cb54140a68ff997944e8d9f72a1c89f2ecb055a6cb6e"} Jul 17 18:20:14.542865 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:14.542832 2574 generic.go:356] "Generic (PLEG): container finished" podID="d530976f-894a-46f3-867d-3a41f32abd96" containerID="9abda98870fdbd71865743b83d2e7c5efcab7c56abc2d303a63753c6b3416f16" exitCode=6 Jul 17 18:20:14.543328 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:14.542916 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8" event={"ID":"d530976f-894a-46f3-867d-3a41f32abd96","Type":"ContainerDied","Data":"9abda98870fdbd71865743b83d2e7c5efcab7c56abc2d303a63753c6b3416f16"} Jul 17 18:20:14.543328 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:14.543076 2574 scope.go:117] "RemoveContainer" containerID="9abda98870fdbd71865743b83d2e7c5efcab7c56abc2d303a63753c6b3416f16" Jul 17 18:20:15.549807 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:15.549775 2574 generic.go:356] "Generic (PLEG): container finished" podID="d530976f-894a-46f3-867d-3a41f32abd96" containerID="4f792cea5c3925837c95746d2a3f2d93264879bcd9d52afa74f815639b1faa5a" exitCode=6 Jul 17 18:20:15.550232 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:15.549868 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8" event={"ID":"d530976f-894a-46f3-867d-3a41f32abd96","Type":"ContainerDied","Data":"4f792cea5c3925837c95746d2a3f2d93264879bcd9d52afa74f815639b1faa5a"} Jul 17 18:20:15.550232 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:15.549923 2574 scope.go:117] "RemoveContainer" containerID="9abda98870fdbd71865743b83d2e7c5efcab7c56abc2d303a63753c6b3416f16" Jul 17 18:20:15.550232 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:15.550062 2574 scope.go:117] "RemoveContainer" containerID="4f792cea5c3925837c95746d2a3f2d93264879bcd9d52afa74f815639b1faa5a" Jul 17 18:20:15.550385 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:20:15.550318 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8_odh-ai-gateway-infra(d530976f-894a-46f3-867d-3a41f32abd96)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8" podUID="d530976f-894a-46f3-867d-3a41f32abd96" Jul 17 18:20:16.557175 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:16.557142 2574 scope.go:117] "RemoveContainer" containerID="4f792cea5c3925837c95746d2a3f2d93264879bcd9d52afa74f815639b1faa5a" Jul 17 18:20:16.557593 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:20:16.557344 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8_odh-ai-gateway-infra(d530976f-894a-46f3-867d-3a41f32abd96)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8" podUID="d530976f-894a-46f3-867d-3a41f32abd96" Jul 17 18:20:29.031413 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:29.031376 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg"] Jul 17 18:20:29.035042 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:29.035023 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg" Jul 17 18:20:29.042935 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:29.042908 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg"] Jul 17 18:20:29.144880 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:29.144835 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c0559bb-033c-438e-a324-0cef9e13fcdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg\" (UID: \"6c0559bb-033c-438e-a324-0cef9e13fcdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg" Jul 17 18:20:29.245550 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:29.245510 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c0559bb-033c-438e-a324-0cef9e13fcdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg\" (UID: \"6c0559bb-033c-438e-a324-0cef9e13fcdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg" Jul 17 18:20:29.246108 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:29.246086 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c0559bb-033c-438e-a324-0cef9e13fcdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg\" (UID: \"6c0559bb-033c-438e-a324-0cef9e13fcdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg" Jul 17 18:20:29.346648 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:29.346558 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg" Jul 17 18:20:29.485283 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:29.485253 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg"] Jul 17 18:20:29.487348 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:20:29.487321 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0559bb_033c_438e_a324_0cef9e13fcdf.slice/crio-af392190aa595160c7d50143449264636a7ef2116b0bcf9db6c38e9b210b8402 WatchSource:0}: Error finding container af392190aa595160c7d50143449264636a7ef2116b0bcf9db6c38e9b210b8402: Status 404 returned error can't find the container with id af392190aa595160c7d50143449264636a7ef2116b0bcf9db6c38e9b210b8402 Jul 17 18:20:29.635174 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:29.635071 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg" event={"ID":"6c0559bb-033c-438e-a324-0cef9e13fcdf","Type":"ContainerStarted","Data":"af392190aa595160c7d50143449264636a7ef2116b0bcf9db6c38e9b210b8402"} Jul 17 18:20:29.979586 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:29.979559 2574 scope.go:117] "RemoveContainer" containerID="4f792cea5c3925837c95746d2a3f2d93264879bcd9d52afa74f815639b1faa5a" Jul 17 18:20:30.642109 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:30.642017 2574 generic.go:356] "Generic (PLEG): container finished" podID="6c0559bb-033c-438e-a324-0cef9e13fcdf" containerID="c4d3f3a79d3677bd5c7bd1923192cc9bbf3b1e3f775d2ffdbaca6bc0dffa096e" exitCode=7 Jul 17 18:20:30.642576 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:30.642114 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg" event={"ID":"6c0559bb-033c-438e-a324-0cef9e13fcdf","Type":"ContainerDied","Data":"c4d3f3a79d3677bd5c7bd1923192cc9bbf3b1e3f775d2ffdbaca6bc0dffa096e"} Jul 17 18:20:30.642576 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:30.642319 2574 scope.go:117] "RemoveContainer" containerID="c4d3f3a79d3677bd5c7bd1923192cc9bbf3b1e3f775d2ffdbaca6bc0dffa096e" Jul 17 18:20:30.644298 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:30.644276 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8" event={"ID":"d530976f-894a-46f3-867d-3a41f32abd96","Type":"ContainerStarted","Data":"f0c8bdd8e8889b2158cfc6d305e10467926328e06f0014c8a13c455c70662c1a"} Jul 17 18:20:30.672321 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:30.672273 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8" podStartSLOduration=18.672257495 podStartE2EDuration="18.672257495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:20:30.670844422 +0000 UTC m=+1842.258797113" watchObservedRunningTime="2026-07-17 18:20:30.672257495 +0000 UTC m=+1842.260210186" Jul 17 18:20:31.651212 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:31.651126 2574 generic.go:356] "Generic (PLEG): container finished" podID="d530976f-894a-46f3-867d-3a41f32abd96" containerID="f0c8bdd8e8889b2158cfc6d305e10467926328e06f0014c8a13c455c70662c1a" exitCode=6 Jul 17 18:20:31.651212 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:31.651201 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8" event={"ID":"d530976f-894a-46f3-867d-3a41f32abd96","Type":"ContainerDied","Data":"f0c8bdd8e8889b2158cfc6d305e10467926328e06f0014c8a13c455c70662c1a"} Jul 17 18:20:31.651694 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:31.651262 2574 scope.go:117] "RemoveContainer" containerID="4f792cea5c3925837c95746d2a3f2d93264879bcd9d52afa74f815639b1faa5a" Jul 17 18:20:31.651694 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:31.651522 2574 scope.go:117] "RemoveContainer" containerID="f0c8bdd8e8889b2158cfc6d305e10467926328e06f0014c8a13c455c70662c1a" Jul 17 18:20:31.651789 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:20:31.651776 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8_odh-ai-gateway-infra(d530976f-894a-46f3-867d-3a41f32abd96)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8" podUID="d530976f-894a-46f3-867d-3a41f32abd96" Jul 17 18:20:31.653042 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:31.653020 2574 generic.go:356] "Generic (PLEG): container finished" podID="6c0559bb-033c-438e-a324-0cef9e13fcdf" containerID="7c9f0f5594c1a7d79d630867fad24b8b27ec9cbd20c9c7de35e941e0f6e1048b" exitCode=7 Jul 17 18:20:31.653136 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:31.653099 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg" event={"ID":"6c0559bb-033c-438e-a324-0cef9e13fcdf","Type":"ContainerDied","Data":"7c9f0f5594c1a7d79d630867fad24b8b27ec9cbd20c9c7de35e941e0f6e1048b"} Jul 17 18:20:31.653249 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:31.653236 2574 scope.go:117] "RemoveContainer" containerID="7c9f0f5594c1a7d79d630867fad24b8b27ec9cbd20c9c7de35e941e0f6e1048b" Jul 17 18:20:31.653456 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:20:31.653440 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg_odh-ai-gateway-infra(6c0559bb-033c-438e-a324-0cef9e13fcdf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg" podUID="6c0559bb-033c-438e-a324-0cef9e13fcdf" Jul 17 18:20:31.662658 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:31.662634 2574 scope.go:117] "RemoveContainer" containerID="c4d3f3a79d3677bd5c7bd1923192cc9bbf3b1e3f775d2ffdbaca6bc0dffa096e" Jul 17 18:20:31.680139 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:31.680109 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8"] Jul 17 18:20:32.660778 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:32.660749 2574 scope.go:117] "RemoveContainer" containerID="7c9f0f5594c1a7d79d630867fad24b8b27ec9cbd20c9c7de35e941e0f6e1048b" Jul 17 18:20:32.661240 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:20:32.661004 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg_odh-ai-gateway-infra(6c0559bb-033c-438e-a324-0cef9e13fcdf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg" podUID="6c0559bb-033c-438e-a324-0cef9e13fcdf" Jul 17 18:20:32.796931 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:32.796905 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8" Jul 17 18:20:32.879982 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:32.879936 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d530976f-894a-46f3-867d-3a41f32abd96-maas-api-service-ca\") pod \"d530976f-894a-46f3-867d-3a41f32abd96\" (UID: \"d530976f-894a-46f3-867d-3a41f32abd96\") " Jul 17 18:20:32.880304 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:32.880281 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d530976f-894a-46f3-867d-3a41f32abd96-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d530976f-894a-46f3-867d-3a41f32abd96" (UID: "d530976f-894a-46f3-867d-3a41f32abd96"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:20:32.981391 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:32.981342 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d530976f-894a-46f3-867d-3a41f32abd96-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:20:33.668140 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:33.668111 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8" Jul 17 18:20:33.668649 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:33.668144 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8" event={"ID":"d530976f-894a-46f3-867d-3a41f32abd96","Type":"ContainerDied","Data":"91c4d177359d0771a940cb54140a68ff997944e8d9f72a1c89f2ecb055a6cb6e"} Jul 17 18:20:33.668649 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:33.668203 2574 scope.go:117] "RemoveContainer" containerID="f0c8bdd8e8889b2158cfc6d305e10467926328e06f0014c8a13c455c70662c1a" Jul 17 18:20:33.683864 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:33.683834 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8"] Jul 17 18:20:33.687559 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:33.687538 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rbkz8"] Jul 17 18:20:34.985470 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:34.985435 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d530976f-894a-46f3-867d-3a41f32abd96" path="/var/lib/kubelet/pods/d530976f-894a-46f3-867d-3a41f32abd96/volumes" Jul 17 18:20:47.980324 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:47.980291 2574 scope.go:117] "RemoveContainer" containerID="7c9f0f5594c1a7d79d630867fad24b8b27ec9cbd20c9c7de35e941e0f6e1048b" Jul 17 18:20:48.747490 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:48.747458 2574 generic.go:356] "Generic (PLEG): container finished" podID="6c0559bb-033c-438e-a324-0cef9e13fcdf" containerID="b2d8b9b71b10822f0960270d0c0d80c5bc182322043b24d130675ad30099e3a1" exitCode=7 Jul 17 18:20:48.747667 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:48.747529 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg" event={"ID":"6c0559bb-033c-438e-a324-0cef9e13fcdf","Type":"ContainerDied","Data":"b2d8b9b71b10822f0960270d0c0d80c5bc182322043b24d130675ad30099e3a1"} Jul 17 18:20:48.747667 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:48.747592 2574 scope.go:117] "RemoveContainer" containerID="7c9f0f5594c1a7d79d630867fad24b8b27ec9cbd20c9c7de35e941e0f6e1048b" Jul 17 18:20:48.747943 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:48.747921 2574 scope.go:117] "RemoveContainer" containerID="b2d8b9b71b10822f0960270d0c0d80c5bc182322043b24d130675ad30099e3a1" Jul 17 18:20:48.748206 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:20:48.748185 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg_odh-ai-gateway-infra(6c0559bb-033c-438e-a324-0cef9e13fcdf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg" podUID="6c0559bb-033c-438e-a324-0cef9e13fcdf" Jul 17 18:20:49.771841 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:49.771802 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg"] Jul 17 18:20:49.906802 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:49.906774 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg" Jul 17 18:20:49.939864 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:49.939825 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c0559bb-033c-438e-a324-0cef9e13fcdf-maas-api-service-ca\") pod \"6c0559bb-033c-438e-a324-0cef9e13fcdf\" (UID: \"6c0559bb-033c-438e-a324-0cef9e13fcdf\") " Jul 17 18:20:49.940311 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:49.940281 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c0559bb-033c-438e-a324-0cef9e13fcdf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6c0559bb-033c-438e-a324-0cef9e13fcdf" (UID: "6c0559bb-033c-438e-a324-0cef9e13fcdf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:20:50.041507 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.041413 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c0559bb-033c-438e-a324-0cef9e13fcdf-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:20:50.614212 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.614179 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-zkj92"] Jul 17 18:20:50.614625 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.614611 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d530976f-894a-46f3-867d-3a41f32abd96" containerName="revoke-keys" Jul 17 18:20:50.614672 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.614628 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d530976f-894a-46f3-867d-3a41f32abd96" containerName="revoke-keys" Jul 17 18:20:50.614672 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.614641 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c0559bb-033c-438e-a324-0cef9e13fcdf" containerName="revoke-keys" Jul 17 18:20:50.614672 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.614649 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c0559bb-033c-438e-a324-0cef9e13fcdf" containerName="revoke-keys" Jul 17 18:20:50.614672 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.614657 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c0559bb-033c-438e-a324-0cef9e13fcdf" containerName="revoke-keys" Jul 17 18:20:50.614672 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.614662 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c0559bb-033c-438e-a324-0cef9e13fcdf" containerName="revoke-keys" Jul 17 18:20:50.614821 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.614674 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c0559bb-033c-438e-a324-0cef9e13fcdf" containerName="revoke-keys" Jul 17 18:20:50.614821 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.614679 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c0559bb-033c-438e-a324-0cef9e13fcdf" containerName="revoke-keys" Jul 17 18:20:50.614821 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.614697 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d530976f-894a-46f3-867d-3a41f32abd96" containerName="revoke-keys" Jul 17 18:20:50.614821 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.614702 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d530976f-894a-46f3-867d-3a41f32abd96" containerName="revoke-keys" Jul 17 18:20:50.614821 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.614768 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d530976f-894a-46f3-867d-3a41f32abd96" containerName="revoke-keys" Jul 17 18:20:50.614821 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.614776 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c0559bb-033c-438e-a324-0cef9e13fcdf" containerName="revoke-keys" Jul 17 18:20:50.614821 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.614782 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c0559bb-033c-438e-a324-0cef9e13fcdf" containerName="revoke-keys" Jul 17 18:20:50.614821 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.614793 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c0559bb-033c-438e-a324-0cef9e13fcdf" containerName="revoke-keys" Jul 17 18:20:50.614821 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.614805 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d530976f-894a-46f3-867d-3a41f32abd96" containerName="revoke-keys" Jul 17 18:20:50.617960 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.617941 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-zkj92" Jul 17 18:20:50.625274 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.625228 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-zkj92"] Jul 17 18:20:50.647632 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.647597 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d258206e-a7c7-416e-86a8-900a894ebb23-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-zkj92\" (UID: \"d258206e-a7c7-416e-86a8-900a894ebb23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-zkj92" Jul 17 18:20:50.748162 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.748126 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d258206e-a7c7-416e-86a8-900a894ebb23-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-zkj92\" (UID: \"d258206e-a7c7-416e-86a8-900a894ebb23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-zkj92" Jul 17 18:20:50.748809 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.748787 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d258206e-a7c7-416e-86a8-900a894ebb23-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-zkj92\" (UID: \"d258206e-a7c7-416e-86a8-900a894ebb23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-zkj92" Jul 17 18:20:50.760454 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.760429 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg" Jul 17 18:20:50.760593 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.760452 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg" event={"ID":"6c0559bb-033c-438e-a324-0cef9e13fcdf","Type":"ContainerDied","Data":"af392190aa595160c7d50143449264636a7ef2116b0bcf9db6c38e9b210b8402"} Jul 17 18:20:50.760593 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.760513 2574 scope.go:117] "RemoveContainer" containerID="b2d8b9b71b10822f0960270d0c0d80c5bc182322043b24d130675ad30099e3a1" Jul 17 18:20:50.783888 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.783844 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg"] Jul 17 18:20:50.789230 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.789205 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-ch8bg"] Jul 17 18:20:50.929764 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.929677 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-zkj92" Jul 17 18:20:50.987918 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:50.987888 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c0559bb-033c-438e-a324-0cef9e13fcdf" path="/var/lib/kubelet/pods/6c0559bb-033c-438e-a324-0cef9e13fcdf/volumes" Jul 17 18:20:51.069458 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:51.069427 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-zkj92"] Jul 17 18:20:51.071649 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:20:51.071618 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd258206e_a7c7_416e_86a8_900a894ebb23.slice/crio-7a74ad5f60ebd9248c3d890a1645ed5509ed8f14cdf046ebe798e83417ba3a72 WatchSource:0}: Error finding container 7a74ad5f60ebd9248c3d890a1645ed5509ed8f14cdf046ebe798e83417ba3a72: Status 404 returned error can't find the container with id 7a74ad5f60ebd9248c3d890a1645ed5509ed8f14cdf046ebe798e83417ba3a72 Jul 17 18:20:51.767064 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:51.767032 2574 generic.go:356] "Generic (PLEG): container finished" podID="d258206e-a7c7-416e-86a8-900a894ebb23" containerID="5799fff1c84a0dc5f8ee562ab9fc87bdef4278604bd2f850f8846aba07e9fb8e" exitCode=6 Jul 17 18:20:51.767250 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:51.767112 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-zkj92" event={"ID":"d258206e-a7c7-416e-86a8-900a894ebb23","Type":"ContainerDied","Data":"5799fff1c84a0dc5f8ee562ab9fc87bdef4278604bd2f850f8846aba07e9fb8e"} Jul 17 18:20:51.767250 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:51.767165 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-zkj92" event={"ID":"d258206e-a7c7-416e-86a8-900a894ebb23","Type":"ContainerStarted","Data":"7a74ad5f60ebd9248c3d890a1645ed5509ed8f14cdf046ebe798e83417ba3a72"} Jul 17 18:20:51.767458 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:51.767441 2574 scope.go:117] "RemoveContainer" containerID="5799fff1c84a0dc5f8ee562ab9fc87bdef4278604bd2f850f8846aba07e9fb8e" Jul 17 18:20:52.774852 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:52.774818 2574 generic.go:356] "Generic (PLEG): container finished" podID="d258206e-a7c7-416e-86a8-900a894ebb23" containerID="582237c1b3c499c945dfbcc672784d63b4d0af105c10039dd6aea0aee11df85a" exitCode=6 Jul 17 18:20:52.775304 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:52.774907 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-zkj92" event={"ID":"d258206e-a7c7-416e-86a8-900a894ebb23","Type":"ContainerDied","Data":"582237c1b3c499c945dfbcc672784d63b4d0af105c10039dd6aea0aee11df85a"} Jul 17 18:20:52.775304 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:52.774976 2574 scope.go:117] "RemoveContainer" containerID="5799fff1c84a0dc5f8ee562ab9fc87bdef4278604bd2f850f8846aba07e9fb8e" Jul 17 18:20:52.775304 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:52.775160 2574 scope.go:117] "RemoveContainer" containerID="582237c1b3c499c945dfbcc672784d63b4d0af105c10039dd6aea0aee11df85a" Jul 17 18:20:52.775482 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:20:52.775469 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-zkj92_odh-ai-gateway-infra(d258206e-a7c7-416e-86a8-900a894ebb23)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-zkj92" podUID="d258206e-a7c7-416e-86a8-900a894ebb23" Jul 17 18:20:53.782595 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:20:53.782562 2574 scope.go:117] "RemoveContainer" containerID="582237c1b3c499c945dfbcc672784d63b4d0af105c10039dd6aea0aee11df85a" Jul 17 18:20:53.782985 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:20:53.782777 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-zkj92_odh-ai-gateway-infra(d258206e-a7c7-416e-86a8-900a894ebb23)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-zkj92" podUID="d258206e-a7c7-416e-86a8-900a894ebb23" Jul 17 18:21:06.979537 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:06.979500 2574 scope.go:117] "RemoveContainer" containerID="582237c1b3c499c945dfbcc672784d63b4d0af105c10039dd6aea0aee11df85a" Jul 17 18:21:07.861859 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:07.861822 2574 generic.go:356] "Generic (PLEG): container finished" podID="d258206e-a7c7-416e-86a8-900a894ebb23" containerID="6a07dbe94453d75af7da639531acb70a18afc5b826d0d353a6a1487f64f34a97" exitCode=6 Jul 17 18:21:07.862042 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:07.861898 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-zkj92" event={"ID":"d258206e-a7c7-416e-86a8-900a894ebb23","Type":"ContainerDied","Data":"6a07dbe94453d75af7da639531acb70a18afc5b826d0d353a6a1487f64f34a97"} Jul 17 18:21:07.862042 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:07.861962 2574 scope.go:117] "RemoveContainer" containerID="582237c1b3c499c945dfbcc672784d63b4d0af105c10039dd6aea0aee11df85a" Jul 17 18:21:07.862274 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:07.862254 2574 scope.go:117] "RemoveContainer" containerID="6a07dbe94453d75af7da639531acb70a18afc5b826d0d353a6a1487f64f34a97" Jul 17 18:21:07.862516 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:21:07.862489 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-zkj92_odh-ai-gateway-infra(d258206e-a7c7-416e-86a8-900a894ebb23)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-zkj92" podUID="d258206e-a7c7-416e-86a8-900a894ebb23" Jul 17 18:21:08.262241 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:08.262209 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:21:08.888510 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:08.888474 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-zkj92"] Jul 17 18:21:09.026234 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:09.026202 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-zkj92" Jul 17 18:21:09.127034 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:09.126968 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d258206e-a7c7-416e-86a8-900a894ebb23-maas-api-service-ca\") pod \"d258206e-a7c7-416e-86a8-900a894ebb23\" (UID: \"d258206e-a7c7-416e-86a8-900a894ebb23\") " Jul 17 18:21:09.127459 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:09.127417 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d258206e-a7c7-416e-86a8-900a894ebb23-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d258206e-a7c7-416e-86a8-900a894ebb23" (UID: "d258206e-a7c7-416e-86a8-900a894ebb23"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:21:09.227953 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:09.227923 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d258206e-a7c7-416e-86a8-900a894ebb23-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:21:09.874103 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:09.874023 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-zkj92" event={"ID":"d258206e-a7c7-416e-86a8-900a894ebb23","Type":"ContainerDied","Data":"7a74ad5f60ebd9248c3d890a1645ed5509ed8f14cdf046ebe798e83417ba3a72"} Jul 17 18:21:09.874103 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:09.874050 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-zkj92" Jul 17 18:21:09.874103 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:09.874090 2574 scope.go:117] "RemoveContainer" containerID="6a07dbe94453d75af7da639531acb70a18afc5b826d0d353a6a1487f64f34a97" Jul 17 18:21:09.895520 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:09.895492 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-zkj92"] Jul 17 18:21:09.901187 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:09.901160 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-zkj92"] Jul 17 18:21:10.985656 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:10.985622 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d258206e-a7c7-416e-86a8-900a894ebb23" path="/var/lib/kubelet/pods/d258206e-a7c7-416e-86a8-900a894ebb23/volumes" Jul 17 18:21:12.841885 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:12.841846 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5"] Jul 17 18:21:12.842274 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:12.842248 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d258206e-a7c7-416e-86a8-900a894ebb23" containerName="revoke-keys" Jul 17 18:21:12.842274 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:12.842260 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d258206e-a7c7-416e-86a8-900a894ebb23" containerName="revoke-keys" Jul 17 18:21:12.842274 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:12.842268 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d258206e-a7c7-416e-86a8-900a894ebb23" containerName="revoke-keys" Jul 17 18:21:12.842274 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:12.842274 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d258206e-a7c7-416e-86a8-900a894ebb23" containerName="revoke-keys" Jul 17 18:21:12.842441 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:12.842288 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d530976f-894a-46f3-867d-3a41f32abd96" containerName="revoke-keys" Jul 17 18:21:12.842441 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:12.842293 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d530976f-894a-46f3-867d-3a41f32abd96" containerName="revoke-keys" Jul 17 18:21:12.842441 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:12.842370 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d258206e-a7c7-416e-86a8-900a894ebb23" containerName="revoke-keys" Jul 17 18:21:12.842441 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:12.842380 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d258206e-a7c7-416e-86a8-900a894ebb23" containerName="revoke-keys" Jul 17 18:21:12.842441 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:12.842394 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d530976f-894a-46f3-867d-3a41f32abd96" containerName="revoke-keys" Jul 17 18:21:12.846936 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:12.846916 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5" Jul 17 18:21:12.849775 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:12.849753 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:21:12.849901 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:12.849756 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dpsm\"" Jul 17 18:21:12.853912 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:12.853884 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5"] Jul 17 18:21:12.962264 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:12.962214 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12592810-e778-4577-a997-db786d29bd0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5\" (UID: \"12592810-e778-4577-a997-db786d29bd0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5" Jul 17 18:21:13.063019 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:13.062973 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12592810-e778-4577-a997-db786d29bd0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5\" (UID: \"12592810-e778-4577-a997-db786d29bd0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5" Jul 17 18:21:13.063690 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:13.063670 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12592810-e778-4577-a997-db786d29bd0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5\" (UID: \"12592810-e778-4577-a997-db786d29bd0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5" Jul 17 18:21:13.160105 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:13.160007 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5" Jul 17 18:21:13.300126 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:13.300103 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5"] Jul 17 18:21:13.302011 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:21:13.301981 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12592810_e778_4577_a997_db786d29bd0f.slice/crio-8a11c3d584dead8422691385c5e7776e8a951e0f94d5ec78f94f13131a97e24b WatchSource:0}: Error finding container 8a11c3d584dead8422691385c5e7776e8a951e0f94d5ec78f94f13131a97e24b: Status 404 returned error can't find the container with id 8a11c3d584dead8422691385c5e7776e8a951e0f94d5ec78f94f13131a97e24b Jul 17 18:21:13.898339 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:13.898297 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5" event={"ID":"12592810-e778-4577-a997-db786d29bd0f","Type":"ContainerStarted","Data":"502be820a2d7a1eea55b92f65674a7992685ab313bad7ce2fe203b05e979226d"} Jul 17 18:21:13.898730 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:13.898374 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5" event={"ID":"12592810-e778-4577-a997-db786d29bd0f","Type":"ContainerStarted","Data":"8a11c3d584dead8422691385c5e7776e8a951e0f94d5ec78f94f13131a97e24b"} Jul 17 18:21:13.898730 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:13.898631 2574 scope.go:117] "RemoveContainer" containerID="502be820a2d7a1eea55b92f65674a7992685ab313bad7ce2fe203b05e979226d" Jul 17 18:21:14.905601 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:14.905508 2574 generic.go:356] "Generic (PLEG): container finished" podID="12592810-e778-4577-a997-db786d29bd0f" containerID="502be820a2d7a1eea55b92f65674a7992685ab313bad7ce2fe203b05e979226d" exitCode=6 Jul 17 18:21:14.905601 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:14.905533 2574 generic.go:356] "Generic (PLEG): container finished" podID="12592810-e778-4577-a997-db786d29bd0f" containerID="9eb41821c254818a2800b019d4881f945660d95ed91dae608e20ec5fd25965fc" exitCode=6 Jul 17 18:21:14.906093 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:14.905603 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5" event={"ID":"12592810-e778-4577-a997-db786d29bd0f","Type":"ContainerDied","Data":"502be820a2d7a1eea55b92f65674a7992685ab313bad7ce2fe203b05e979226d"} Jul 17 18:21:14.906093 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:14.905663 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5" event={"ID":"12592810-e778-4577-a997-db786d29bd0f","Type":"ContainerDied","Data":"9eb41821c254818a2800b019d4881f945660d95ed91dae608e20ec5fd25965fc"} Jul 17 18:21:14.906093 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:14.905699 2574 scope.go:117] "RemoveContainer" containerID="502be820a2d7a1eea55b92f65674a7992685ab313bad7ce2fe203b05e979226d" Jul 17 18:21:14.906093 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:14.905754 2574 scope.go:117] "RemoveContainer" containerID="9eb41821c254818a2800b019d4881f945660d95ed91dae608e20ec5fd25965fc" Jul 17 18:21:14.906093 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:21:14.906014 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5_odh-ai-gateway-infra(12592810-e778-4577-a997-db786d29bd0f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5" podUID="12592810-e778-4577-a997-db786d29bd0f" Jul 17 18:21:14.917393 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:14.917365 2574 scope.go:117] "RemoveContainer" containerID="502be820a2d7a1eea55b92f65674a7992685ab313bad7ce2fe203b05e979226d" Jul 17 18:21:14.917725 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:21:14.917703 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"502be820a2d7a1eea55b92f65674a7992685ab313bad7ce2fe203b05e979226d\": container with ID starting with 502be820a2d7a1eea55b92f65674a7992685ab313bad7ce2fe203b05e979226d not found: ID does not exist" containerID="502be820a2d7a1eea55b92f65674a7992685ab313bad7ce2fe203b05e979226d" Jul 17 18:21:14.917781 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:14.917737 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"502be820a2d7a1eea55b92f65674a7992685ab313bad7ce2fe203b05e979226d"} err="failed to get container status \"502be820a2d7a1eea55b92f65674a7992685ab313bad7ce2fe203b05e979226d\": rpc error: code = NotFound desc = could not find container \"502be820a2d7a1eea55b92f65674a7992685ab313bad7ce2fe203b05e979226d\": container with ID starting with 502be820a2d7a1eea55b92f65674a7992685ab313bad7ce2fe203b05e979226d not found: ID does not exist" Jul 17 18:21:15.912303 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:15.912273 2574 scope.go:117] "RemoveContainer" containerID="9eb41821c254818a2800b019d4881f945660d95ed91dae608e20ec5fd25965fc" Jul 17 18:21:15.912795 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:21:15.912623 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5_odh-ai-gateway-infra(12592810-e778-4577-a997-db786d29bd0f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5" podUID="12592810-e778-4577-a997-db786d29bd0f" Jul 17 18:21:26.980234 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:26.980200 2574 scope.go:117] "RemoveContainer" containerID="9eb41821c254818a2800b019d4881f945660d95ed91dae608e20ec5fd25965fc" Jul 17 18:21:27.977452 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:27.977410 2574 generic.go:356] "Generic (PLEG): container finished" podID="12592810-e778-4577-a997-db786d29bd0f" containerID="cc3cabe2b874672975f324937137e246f01b890d7485f09e6443c3fa107740b6" exitCode=6 Jul 17 18:21:27.977664 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:27.977492 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5" event={"ID":"12592810-e778-4577-a997-db786d29bd0f","Type":"ContainerDied","Data":"cc3cabe2b874672975f324937137e246f01b890d7485f09e6443c3fa107740b6"} Jul 17 18:21:27.977664 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:27.977565 2574 scope.go:117] "RemoveContainer" containerID="9eb41821c254818a2800b019d4881f945660d95ed91dae608e20ec5fd25965fc" Jul 17 18:21:27.977939 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:27.977918 2574 scope.go:117] "RemoveContainer" containerID="cc3cabe2b874672975f324937137e246f01b890d7485f09e6443c3fa107740b6" Jul 17 18:21:27.978180 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:21:27.978157 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5_odh-ai-gateway-infra(12592810-e778-4577-a997-db786d29bd0f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5" podUID="12592810-e778-4577-a997-db786d29bd0f" Jul 17 18:21:29.005315 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:29.005276 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5"] Jul 17 18:21:29.075581 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:29.075541 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl"] Jul 17 18:21:29.076174 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:29.076154 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d258206e-a7c7-416e-86a8-900a894ebb23" containerName="revoke-keys" Jul 17 18:21:29.076174 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:29.076176 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d258206e-a7c7-416e-86a8-900a894ebb23" containerName="revoke-keys" Jul 17 18:21:29.076336 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:29.076304 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d258206e-a7c7-416e-86a8-900a894ebb23" containerName="revoke-keys" Jul 17 18:21:29.081157 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:29.081131 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl" Jul 17 18:21:29.085581 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:29.085505 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl"] Jul 17 18:21:29.117498 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:29.117454 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9cf0fe-68c4-4dc0-9739-8321f6a27048-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl\" (UID: \"4c9cf0fe-68c4-4dc0-9739-8321f6a27048\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl" Jul 17 18:21:29.169855 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:29.169820 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5" Jul 17 18:21:29.218638 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:29.218592 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12592810-e778-4577-a997-db786d29bd0f-maas-api-service-ca\") pod \"12592810-e778-4577-a997-db786d29bd0f\" (UID: \"12592810-e778-4577-a997-db786d29bd0f\") " Jul 17 18:21:29.218849 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:29.218825 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9cf0fe-68c4-4dc0-9739-8321f6a27048-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl\" (UID: \"4c9cf0fe-68c4-4dc0-9739-8321f6a27048\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl" Jul 17 18:21:29.218978 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:29.218954 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12592810-e778-4577-a997-db786d29bd0f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "12592810-e778-4577-a997-db786d29bd0f" (UID: "12592810-e778-4577-a997-db786d29bd0f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:21:29.219463 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:29.219444 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9cf0fe-68c4-4dc0-9739-8321f6a27048-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl\" (UID: \"4c9cf0fe-68c4-4dc0-9739-8321f6a27048\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl" Jul 17 18:21:29.319978 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:29.319885 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12592810-e778-4577-a997-db786d29bd0f-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:21:29.403892 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:29.403846 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl" Jul 17 18:21:29.546775 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:29.546744 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl"] Jul 17 18:21:29.548891 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:21:29.548863 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9cf0fe_68c4_4dc0_9739_8321f6a27048.slice/crio-8e00d8097d9135f27d8151128e3da3080591a47112d57a6b55f5e56ce09b33ce WatchSource:0}: Error finding container 8e00d8097d9135f27d8151128e3da3080591a47112d57a6b55f5e56ce09b33ce: Status 404 returned error can't find the container with id 8e00d8097d9135f27d8151128e3da3080591a47112d57a6b55f5e56ce09b33ce Jul 17 18:21:29.990988 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:29.990956 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5" Jul 17 18:21:29.991190 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:29.990956 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5" event={"ID":"12592810-e778-4577-a997-db786d29bd0f","Type":"ContainerDied","Data":"8a11c3d584dead8422691385c5e7776e8a951e0f94d5ec78f94f13131a97e24b"} Jul 17 18:21:29.991190 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:29.991095 2574 scope.go:117] "RemoveContainer" containerID="cc3cabe2b874672975f324937137e246f01b890d7485f09e6443c3fa107740b6" Jul 17 18:21:29.992646 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:29.992618 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl" event={"ID":"4c9cf0fe-68c4-4dc0-9739-8321f6a27048","Type":"ContainerStarted","Data":"30476be948c1665eaa1ad65c6981e21b3538f97ed3d4efbf7fb36b92c78c0686"} Jul 17 18:21:29.992748 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:29.992675 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl" event={"ID":"4c9cf0fe-68c4-4dc0-9739-8321f6a27048","Type":"ContainerStarted","Data":"8e00d8097d9135f27d8151128e3da3080591a47112d57a6b55f5e56ce09b33ce"} Jul 17 18:21:30.013461 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:30.013411 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl" podStartSLOduration=1.0133950330000001 podStartE2EDuration="1.013395033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:21:30.011082102 +0000 UTC m=+1901.599034792" watchObservedRunningTime="2026-07-17 18:21:30.013395033 +0000 UTC m=+1901.601347723" Jul 17 18:21:30.026225 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:30.026180 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5"] Jul 17 18:21:30.030302 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:30.030274 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-96hj5"] Jul 17 18:21:30.989200 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:30.989162 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12592810-e778-4577-a997-db786d29bd0f" path="/var/lib/kubelet/pods/12592810-e778-4577-a997-db786d29bd0f/volumes" Jul 17 18:21:31.000282 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:31.000250 2574 generic.go:356] "Generic (PLEG): container finished" podID="4c9cf0fe-68c4-4dc0-9739-8321f6a27048" containerID="30476be948c1665eaa1ad65c6981e21b3538f97ed3d4efbf7fb36b92c78c0686" exitCode=7 Jul 17 18:21:31.000448 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:31.000304 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl" event={"ID":"4c9cf0fe-68c4-4dc0-9739-8321f6a27048","Type":"ContainerDied","Data":"30476be948c1665eaa1ad65c6981e21b3538f97ed3d4efbf7fb36b92c78c0686"} Jul 17 18:21:31.000593 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:31.000580 2574 scope.go:117] "RemoveContainer" containerID="30476be948c1665eaa1ad65c6981e21b3538f97ed3d4efbf7fb36b92c78c0686" Jul 17 18:21:32.007927 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:32.007894 2574 generic.go:356] "Generic (PLEG): container finished" podID="4c9cf0fe-68c4-4dc0-9739-8321f6a27048" containerID="0c59695f771ee60bba92983d34eb616f67e4c49d217334c72937cc4f9d8f4121" exitCode=7 Jul 17 18:21:32.008318 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:32.007981 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl" event={"ID":"4c9cf0fe-68c4-4dc0-9739-8321f6a27048","Type":"ContainerDied","Data":"0c59695f771ee60bba92983d34eb616f67e4c49d217334c72937cc4f9d8f4121"} Jul 17 18:21:32.008318 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:32.008044 2574 scope.go:117] "RemoveContainer" containerID="30476be948c1665eaa1ad65c6981e21b3538f97ed3d4efbf7fb36b92c78c0686" Jul 17 18:21:32.008318 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:32.008317 2574 scope.go:117] "RemoveContainer" containerID="0c59695f771ee60bba92983d34eb616f67e4c49d217334c72937cc4f9d8f4121" Jul 17 18:21:32.008584 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:21:32.008559 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl_odh-ai-gateway-infra(4c9cf0fe-68c4-4dc0-9739-8321f6a27048)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl" podUID="4c9cf0fe-68c4-4dc0-9739-8321f6a27048" Jul 17 18:21:33.015477 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:33.015451 2574 scope.go:117] "RemoveContainer" containerID="0c59695f771ee60bba92983d34eb616f67e4c49d217334c72937cc4f9d8f4121" Jul 17 18:21:33.017952 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:21:33.015646 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl_odh-ai-gateway-infra(4c9cf0fe-68c4-4dc0-9739-8321f6a27048)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl" podUID="4c9cf0fe-68c4-4dc0-9739-8321f6a27048" Jul 17 18:21:40.650985 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:40.650953 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz"] Jul 17 18:21:40.651515 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:40.651391 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12592810-e778-4577-a997-db786d29bd0f" containerName="revoke-keys" Jul 17 18:21:40.651515 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:40.651404 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="12592810-e778-4577-a997-db786d29bd0f" containerName="revoke-keys" Jul 17 18:21:40.651515 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:40.651431 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12592810-e778-4577-a997-db786d29bd0f" containerName="revoke-keys" Jul 17 18:21:40.651515 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:40.651436 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="12592810-e778-4577-a997-db786d29bd0f" containerName="revoke-keys" Jul 17 18:21:40.651515 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:40.651494 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="12592810-e778-4577-a997-db786d29bd0f" containerName="revoke-keys" Jul 17 18:21:40.651515 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:40.651501 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="12592810-e778-4577-a997-db786d29bd0f" containerName="revoke-keys" Jul 17 18:21:40.651515 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:40.651510 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="12592810-e778-4577-a997-db786d29bd0f" containerName="revoke-keys" Jul 17 18:21:40.654570 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:40.654551 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz" Jul 17 18:21:40.661389 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:40.661315 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz"] Jul 17 18:21:40.732219 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:40.732178 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73de9986-0808-4f9f-aa3f-f7f0b8e38bca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz\" (UID: \"73de9986-0808-4f9f-aa3f-f7f0b8e38bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz" Jul 17 18:21:40.832777 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:40.832736 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73de9986-0808-4f9f-aa3f-f7f0b8e38bca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz\" (UID: \"73de9986-0808-4f9f-aa3f-f7f0b8e38bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz" Jul 17 18:21:40.838383 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:40.834025 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73de9986-0808-4f9f-aa3f-f7f0b8e38bca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz\" (UID: \"73de9986-0808-4f9f-aa3f-f7f0b8e38bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz" Jul 17 18:21:40.966848 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:40.966809 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz" Jul 17 18:21:41.112051 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:41.112020 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz"] Jul 17 18:21:41.114224 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:21:41.114182 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73de9986_0808_4f9f_aa3f_f7f0b8e38bca.slice/crio-ced99cd287eb7302604ceab85b66e7c00e479e3e2875b2c5d7b2bc0e6f20b7e2 WatchSource:0}: Error finding container ced99cd287eb7302604ceab85b66e7c00e479e3e2875b2c5d7b2bc0e6f20b7e2: Status 404 returned error can't find the container with id ced99cd287eb7302604ceab85b66e7c00e479e3e2875b2c5d7b2bc0e6f20b7e2 Jul 17 18:21:42.062028 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:42.061994 2574 generic.go:356] "Generic (PLEG): container finished" podID="73de9986-0808-4f9f-aa3f-f7f0b8e38bca" containerID="c36ec14583878aa165270dd1f70a9654de3064ca1b2f7272aac90592fd35b3dd" exitCode=6 Jul 17 18:21:42.062439 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:42.062063 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz" event={"ID":"73de9986-0808-4f9f-aa3f-f7f0b8e38bca","Type":"ContainerDied","Data":"c36ec14583878aa165270dd1f70a9654de3064ca1b2f7272aac90592fd35b3dd"} Jul 17 18:21:42.062439 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:42.062102 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz" event={"ID":"73de9986-0808-4f9f-aa3f-f7f0b8e38bca","Type":"ContainerStarted","Data":"ced99cd287eb7302604ceab85b66e7c00e479e3e2875b2c5d7b2bc0e6f20b7e2"} Jul 17 18:21:42.062439 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:42.062248 2574 scope.go:117] "RemoveContainer" containerID="c36ec14583878aa165270dd1f70a9654de3064ca1b2f7272aac90592fd35b3dd" Jul 17 18:21:43.069014 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:43.068977 2574 generic.go:356] "Generic (PLEG): container finished" podID="73de9986-0808-4f9f-aa3f-f7f0b8e38bca" containerID="4bffffcd0ce6a1ede0102c87b8b25167c0b42d537160edda1dea16cf237bc65e" exitCode=6 Jul 17 18:21:43.069466 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:43.069062 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz" event={"ID":"73de9986-0808-4f9f-aa3f-f7f0b8e38bca","Type":"ContainerDied","Data":"4bffffcd0ce6a1ede0102c87b8b25167c0b42d537160edda1dea16cf237bc65e"} Jul 17 18:21:43.069466 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:43.069128 2574 scope.go:117] "RemoveContainer" containerID="c36ec14583878aa165270dd1f70a9654de3064ca1b2f7272aac90592fd35b3dd" Jul 17 18:21:43.069559 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:43.069477 2574 scope.go:117] "RemoveContainer" containerID="4bffffcd0ce6a1ede0102c87b8b25167c0b42d537160edda1dea16cf237bc65e" Jul 17 18:21:43.069759 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:21:43.069738 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz_odh-ai-gateway-infra(73de9986-0808-4f9f-aa3f-f7f0b8e38bca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz" podUID="73de9986-0808-4f9f-aa3f-f7f0b8e38bca" Jul 17 18:21:44.076251 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:44.076224 2574 scope.go:117] "RemoveContainer" containerID="4bffffcd0ce6a1ede0102c87b8b25167c0b42d537160edda1dea16cf237bc65e" Jul 17 18:21:44.076658 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:21:44.076418 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz_odh-ai-gateway-infra(73de9986-0808-4f9f-aa3f-f7f0b8e38bca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz" podUID="73de9986-0808-4f9f-aa3f-f7f0b8e38bca" Jul 17 18:21:47.980040 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:47.980005 2574 scope.go:117] "RemoveContainer" containerID="0c59695f771ee60bba92983d34eb616f67e4c49d217334c72937cc4f9d8f4121" Jul 17 18:21:49.103801 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:49.103764 2574 generic.go:356] "Generic (PLEG): container finished" podID="4c9cf0fe-68c4-4dc0-9739-8321f6a27048" containerID="70ba063d3034b5e306e7f66201e71115054ded98152a71df4fd3f10bc8542138" exitCode=7 Jul 17 18:21:49.104233 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:49.103835 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl" event={"ID":"4c9cf0fe-68c4-4dc0-9739-8321f6a27048","Type":"ContainerDied","Data":"70ba063d3034b5e306e7f66201e71115054ded98152a71df4fd3f10bc8542138"} Jul 17 18:21:49.104233 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:49.103893 2574 scope.go:117] "RemoveContainer" containerID="0c59695f771ee60bba92983d34eb616f67e4c49d217334c72937cc4f9d8f4121" Jul 17 18:21:49.104233 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:49.104205 2574 scope.go:117] "RemoveContainer" containerID="70ba063d3034b5e306e7f66201e71115054ded98152a71df4fd3f10bc8542138" Jul 17 18:21:49.104466 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:21:49.104447 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl_odh-ai-gateway-infra(4c9cf0fe-68c4-4dc0-9739-8321f6a27048)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl" podUID="4c9cf0fe-68c4-4dc0-9739-8321f6a27048" Jul 17 18:21:50.129218 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:50.129180 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl"] Jul 17 18:21:50.256922 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:50.256897 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl" Jul 17 18:21:50.427237 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:50.427147 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9cf0fe-68c4-4dc0-9739-8321f6a27048-maas-api-service-ca\") pod \"4c9cf0fe-68c4-4dc0-9739-8321f6a27048\" (UID: \"4c9cf0fe-68c4-4dc0-9739-8321f6a27048\") " Jul 17 18:21:50.427545 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:50.427524 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9cf0fe-68c4-4dc0-9739-8321f6a27048-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4c9cf0fe-68c4-4dc0-9739-8321f6a27048" (UID: "4c9cf0fe-68c4-4dc0-9739-8321f6a27048"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:21:50.528553 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:50.528510 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c9cf0fe-68c4-4dc0-9739-8321f6a27048-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:21:51.116566 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:51.116515 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl" event={"ID":"4c9cf0fe-68c4-4dc0-9739-8321f6a27048","Type":"ContainerDied","Data":"8e00d8097d9135f27d8151128e3da3080591a47112d57a6b55f5e56ce09b33ce"} Jul 17 18:21:51.116784 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:51.116610 2574 scope.go:117] "RemoveContainer" containerID="70ba063d3034b5e306e7f66201e71115054ded98152a71df4fd3f10bc8542138" Jul 17 18:21:51.116784 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:51.116536 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl" Jul 17 18:21:51.135642 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:51.135609 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl"] Jul 17 18:21:51.139628 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:51.139597 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-h7hvl"] Jul 17 18:21:52.986079 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:52.986043 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c9cf0fe-68c4-4dc0-9739-8321f6a27048" path="/var/lib/kubelet/pods/4c9cf0fe-68c4-4dc0-9739-8321f6a27048/volumes" Jul 17 18:21:54.979483 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:54.979445 2574 scope.go:117] "RemoveContainer" containerID="4bffffcd0ce6a1ede0102c87b8b25167c0b42d537160edda1dea16cf237bc65e" Jul 17 18:21:56.145857 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:56.145820 2574 generic.go:356] "Generic (PLEG): container finished" podID="73de9986-0808-4f9f-aa3f-f7f0b8e38bca" containerID="e7ff866b684dbf6c5d9d9e7c670b9e487c8eb0088bfb176dd5c5abdfe128ebe1" exitCode=6 Jul 17 18:21:56.146281 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:56.145895 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz" event={"ID":"73de9986-0808-4f9f-aa3f-f7f0b8e38bca","Type":"ContainerDied","Data":"e7ff866b684dbf6c5d9d9e7c670b9e487c8eb0088bfb176dd5c5abdfe128ebe1"} Jul 17 18:21:56.146281 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:56.145984 2574 scope.go:117] "RemoveContainer" containerID="4bffffcd0ce6a1ede0102c87b8b25167c0b42d537160edda1dea16cf237bc65e" Jul 17 18:21:56.146407 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:56.146346 2574 scope.go:117] "RemoveContainer" containerID="e7ff866b684dbf6c5d9d9e7c670b9e487c8eb0088bfb176dd5c5abdfe128ebe1" Jul 17 18:21:56.146677 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:21:56.146653 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz_odh-ai-gateway-infra(73de9986-0808-4f9f-aa3f-f7f0b8e38bca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz" podUID="73de9986-0808-4f9f-aa3f-f7f0b8e38bca" Jul 17 18:21:57.169464 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:57.169429 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz"] Jul 17 18:21:57.304629 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:57.304598 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz" Jul 17 18:21:57.494927 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:57.494887 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73de9986-0808-4f9f-aa3f-f7f0b8e38bca-maas-api-service-ca\") pod \"73de9986-0808-4f9f-aa3f-f7f0b8e38bca\" (UID: \"73de9986-0808-4f9f-aa3f-f7f0b8e38bca\") " Jul 17 18:21:57.495300 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:57.495272 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73de9986-0808-4f9f-aa3f-f7f0b8e38bca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "73de9986-0808-4f9f-aa3f-f7f0b8e38bca" (UID: "73de9986-0808-4f9f-aa3f-f7f0b8e38bca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:21:57.595861 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:57.595815 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73de9986-0808-4f9f-aa3f-f7f0b8e38bca-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:21:58.160949 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:58.160915 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz" event={"ID":"73de9986-0808-4f9f-aa3f-f7f0b8e38bca","Type":"ContainerDied","Data":"ced99cd287eb7302604ceab85b66e7c00e479e3e2875b2c5d7b2bc0e6f20b7e2"} Jul 17 18:21:58.161143 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:58.160963 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz" Jul 17 18:21:58.161143 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:58.160980 2574 scope.go:117] "RemoveContainer" containerID="e7ff866b684dbf6c5d9d9e7c670b9e487c8eb0088bfb176dd5c5abdfe128ebe1" Jul 17 18:21:58.188319 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:58.188269 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz"] Jul 17 18:21:58.190239 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:58.190217 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-vv2vz"] Jul 17 18:21:58.986242 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:21:58.986206 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73de9986-0808-4f9f-aa3f-f7f0b8e38bca" path="/var/lib/kubelet/pods/73de9986-0808-4f9f-aa3f-f7f0b8e38bca/volumes" Jul 17 18:22:02.879783 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.879738 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm"] Jul 17 18:22:02.880222 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.880208 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12592810-e778-4577-a997-db786d29bd0f" containerName="revoke-keys" Jul 17 18:22:02.880281 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.880224 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="12592810-e778-4577-a997-db786d29bd0f" containerName="revoke-keys" Jul 17 18:22:02.880281 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.880237 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c9cf0fe-68c4-4dc0-9739-8321f6a27048" containerName="revoke-keys" Jul 17 18:22:02.880281 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.880244 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c9cf0fe-68c4-4dc0-9739-8321f6a27048" containerName="revoke-keys" Jul 17 18:22:02.880281 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.880250 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c9cf0fe-68c4-4dc0-9739-8321f6a27048" containerName="revoke-keys" Jul 17 18:22:02.880281 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.880255 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c9cf0fe-68c4-4dc0-9739-8321f6a27048" containerName="revoke-keys" Jul 17 18:22:02.880281 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.880270 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73de9986-0808-4f9f-aa3f-f7f0b8e38bca" containerName="revoke-keys" Jul 17 18:22:02.880281 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.880276 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="73de9986-0808-4f9f-aa3f-f7f0b8e38bca" containerName="revoke-keys" Jul 17 18:22:02.880281 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.880281 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73de9986-0808-4f9f-aa3f-f7f0b8e38bca" containerName="revoke-keys" Jul 17 18:22:02.880565 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.880286 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="73de9986-0808-4f9f-aa3f-f7f0b8e38bca" containerName="revoke-keys" Jul 17 18:22:02.880565 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.880297 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c9cf0fe-68c4-4dc0-9739-8321f6a27048" containerName="revoke-keys" Jul 17 18:22:02.880565 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.880303 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c9cf0fe-68c4-4dc0-9739-8321f6a27048" containerName="revoke-keys" Jul 17 18:22:02.880565 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.880314 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73de9986-0808-4f9f-aa3f-f7f0b8e38bca" containerName="revoke-keys" Jul 17 18:22:02.880565 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.880321 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="73de9986-0808-4f9f-aa3f-f7f0b8e38bca" containerName="revoke-keys" Jul 17 18:22:02.880565 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.880405 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c9cf0fe-68c4-4dc0-9739-8321f6a27048" containerName="revoke-keys" Jul 17 18:22:02.880565 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.880414 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="73de9986-0808-4f9f-aa3f-f7f0b8e38bca" containerName="revoke-keys" Jul 17 18:22:02.880565 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.880420 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="73de9986-0808-4f9f-aa3f-f7f0b8e38bca" containerName="revoke-keys" Jul 17 18:22:02.880565 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.880427 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c9cf0fe-68c4-4dc0-9739-8321f6a27048" containerName="revoke-keys" Jul 17 18:22:02.884959 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.884936 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm" Jul 17 18:22:02.889103 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.889083 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:22:02.891946 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.891799 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dpsm\"" Jul 17 18:22:02.898253 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.898223 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm"] Jul 17 18:22:02.943047 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:02.943004 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66e96f9-8859-4152-9e30-ecaa14b45151-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm\" (UID: \"d66e96f9-8859-4152-9e30-ecaa14b45151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm" Jul 17 18:22:03.044624 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:03.044585 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66e96f9-8859-4152-9e30-ecaa14b45151-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm\" (UID: \"d66e96f9-8859-4152-9e30-ecaa14b45151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm" Jul 17 18:22:03.045270 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:03.045245 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66e96f9-8859-4152-9e30-ecaa14b45151-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm\" (UID: \"d66e96f9-8859-4152-9e30-ecaa14b45151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm" Jul 17 18:22:03.195884 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:03.195793 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm" Jul 17 18:22:03.341173 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:03.341139 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm"] Jul 17 18:22:03.342202 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:22:03.342170 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66e96f9_8859_4152_9e30_ecaa14b45151.slice/crio-754ecc8ba1c242d380b1860cdb8a52527c227d9259259e843f01d4bbb1e46b87 WatchSource:0}: Error finding container 754ecc8ba1c242d380b1860cdb8a52527c227d9259259e843f01d4bbb1e46b87: Status 404 returned error can't find the container with id 754ecc8ba1c242d380b1860cdb8a52527c227d9259259e843f01d4bbb1e46b87 Jul 17 18:22:04.194132 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:04.194094 2574 generic.go:356] "Generic (PLEG): container finished" podID="d66e96f9-8859-4152-9e30-ecaa14b45151" containerID="0793a9d197dc70499d9e2caeda13896c627dfaa4de26c5b06bb354909ddf6d78" exitCode=6 Jul 17 18:22:04.194615 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:04.194182 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm" event={"ID":"d66e96f9-8859-4152-9e30-ecaa14b45151","Type":"ContainerDied","Data":"0793a9d197dc70499d9e2caeda13896c627dfaa4de26c5b06bb354909ddf6d78"} Jul 17 18:22:04.194615 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:04.194235 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm" event={"ID":"d66e96f9-8859-4152-9e30-ecaa14b45151","Type":"ContainerStarted","Data":"754ecc8ba1c242d380b1860cdb8a52527c227d9259259e843f01d4bbb1e46b87"} Jul 17 18:22:04.194615 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:04.194392 2574 scope.go:117] "RemoveContainer" containerID="0793a9d197dc70499d9e2caeda13896c627dfaa4de26c5b06bb354909ddf6d78" Jul 17 18:22:05.201896 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:05.201800 2574 generic.go:356] "Generic (PLEG): container finished" podID="d66e96f9-8859-4152-9e30-ecaa14b45151" containerID="078706f03d1ac764f1657a5cf23adc9386089b39b81dfe25d2493858fd545ae8" exitCode=6 Jul 17 18:22:05.201896 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:05.201881 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm" event={"ID":"d66e96f9-8859-4152-9e30-ecaa14b45151","Type":"ContainerDied","Data":"078706f03d1ac764f1657a5cf23adc9386089b39b81dfe25d2493858fd545ae8"} Jul 17 18:22:05.202319 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:05.201940 2574 scope.go:117] "RemoveContainer" containerID="0793a9d197dc70499d9e2caeda13896c627dfaa4de26c5b06bb354909ddf6d78" Jul 17 18:22:05.202319 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:05.202180 2574 scope.go:117] "RemoveContainer" containerID="078706f03d1ac764f1657a5cf23adc9386089b39b81dfe25d2493858fd545ae8" Jul 17 18:22:05.202456 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:22:05.202437 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm_odh-ai-gateway-infra(d66e96f9-8859-4152-9e30-ecaa14b45151)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm" podUID="d66e96f9-8859-4152-9e30-ecaa14b45151" Jul 17 18:22:06.209553 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:06.209524 2574 scope.go:117] "RemoveContainer" containerID="078706f03d1ac764f1657a5cf23adc9386089b39b81dfe25d2493858fd545ae8" Jul 17 18:22:06.209943 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:22:06.209709 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm_odh-ai-gateway-infra(d66e96f9-8859-4152-9e30-ecaa14b45151)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm" podUID="d66e96f9-8859-4152-9e30-ecaa14b45151" Jul 17 18:22:16.979464 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:16.979429 2574 scope.go:117] "RemoveContainer" containerID="078706f03d1ac764f1657a5cf23adc9386089b39b81dfe25d2493858fd545ae8" Jul 17 18:22:18.272268 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:18.272237 2574 generic.go:356] "Generic (PLEG): container finished" podID="d66e96f9-8859-4152-9e30-ecaa14b45151" containerID="d969836602f2a8b7f463446ecd2f79970bdc976106f1fb96134d93ee87ce1dd6" exitCode=6 Jul 17 18:22:18.272733 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:18.272304 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm" event={"ID":"d66e96f9-8859-4152-9e30-ecaa14b45151","Type":"ContainerDied","Data":"d969836602f2a8b7f463446ecd2f79970bdc976106f1fb96134d93ee87ce1dd6"} Jul 17 18:22:18.272733 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:18.272388 2574 scope.go:117] "RemoveContainer" containerID="078706f03d1ac764f1657a5cf23adc9386089b39b81dfe25d2493858fd545ae8" Jul 17 18:22:18.272733 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:18.272664 2574 scope.go:117] "RemoveContainer" containerID="d969836602f2a8b7f463446ecd2f79970bdc976106f1fb96134d93ee87ce1dd6" Jul 17 18:22:18.272910 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:22:18.272887 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm_odh-ai-gateway-infra(d66e96f9-8859-4152-9e30-ecaa14b45151)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm" podUID="d66e96f9-8859-4152-9e30-ecaa14b45151" Jul 17 18:22:19.295981 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:19.295949 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm"] Jul 17 18:22:19.437403 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:19.437376 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm" Jul 17 18:22:19.499935 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:19.499904 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66e96f9-8859-4152-9e30-ecaa14b45151-maas-api-service-ca\") pod \"d66e96f9-8859-4152-9e30-ecaa14b45151\" (UID: \"d66e96f9-8859-4152-9e30-ecaa14b45151\") " Jul 17 18:22:19.500298 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:19.500271 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d66e96f9-8859-4152-9e30-ecaa14b45151-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d66e96f9-8859-4152-9e30-ecaa14b45151" (UID: "d66e96f9-8859-4152-9e30-ecaa14b45151"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:22:19.600998 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:19.600896 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d66e96f9-8859-4152-9e30-ecaa14b45151-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:22:20.285466 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:20.285425 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm" event={"ID":"d66e96f9-8859-4152-9e30-ecaa14b45151","Type":"ContainerDied","Data":"754ecc8ba1c242d380b1860cdb8a52527c227d9259259e843f01d4bbb1e46b87"} Jul 17 18:22:20.285466 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:20.285466 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm" Jul 17 18:22:20.285747 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:20.285493 2574 scope.go:117] "RemoveContainer" containerID="d969836602f2a8b7f463446ecd2f79970bdc976106f1fb96134d93ee87ce1dd6" Jul 17 18:22:20.308211 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:20.308168 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm"] Jul 17 18:22:20.313372 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:20.313319 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-zmlfm"] Jul 17 18:22:20.985311 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:20.985273 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d66e96f9-8859-4152-9e30-ecaa14b45151" path="/var/lib/kubelet/pods/d66e96f9-8859-4152-9e30-ecaa14b45151/volumes" Jul 17 18:22:29.111669 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:29.111627 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt"] Jul 17 18:22:29.112254 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:29.112232 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66e96f9-8859-4152-9e30-ecaa14b45151" containerName="revoke-keys" Jul 17 18:22:29.112334 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:29.112257 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66e96f9-8859-4152-9e30-ecaa14b45151" containerName="revoke-keys" Jul 17 18:22:29.112334 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:29.112274 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66e96f9-8859-4152-9e30-ecaa14b45151" containerName="revoke-keys" Jul 17 18:22:29.112334 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:29.112283 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66e96f9-8859-4152-9e30-ecaa14b45151" containerName="revoke-keys" Jul 17 18:22:29.112334 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:29.112300 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66e96f9-8859-4152-9e30-ecaa14b45151" containerName="revoke-keys" Jul 17 18:22:29.112334 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:29.112309 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66e96f9-8859-4152-9e30-ecaa14b45151" containerName="revoke-keys" Jul 17 18:22:29.112612 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:29.112417 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="73de9986-0808-4f9f-aa3f-f7f0b8e38bca" containerName="revoke-keys" Jul 17 18:22:29.112612 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:29.112430 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66e96f9-8859-4152-9e30-ecaa14b45151" containerName="revoke-keys" Jul 17 18:22:29.112612 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:29.112443 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c9cf0fe-68c4-4dc0-9739-8321f6a27048" containerName="revoke-keys" Jul 17 18:22:29.112612 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:29.112454 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66e96f9-8859-4152-9e30-ecaa14b45151" containerName="revoke-keys" Jul 17 18:22:29.115815 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:29.115791 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt" Jul 17 18:22:29.119567 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:29.119542 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dpsm\"" Jul 17 18:22:29.119832 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:29.119627 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:22:29.121519 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:29.121497 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt"] Jul 17 18:22:29.191193 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:29.191150 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/584c95b4-0b10-4564-804e-f40f45f79d89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt\" (UID: \"584c95b4-0b10-4564-804e-f40f45f79d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt" Jul 17 18:22:29.291872 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:29.291832 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/584c95b4-0b10-4564-804e-f40f45f79d89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt\" (UID: \"584c95b4-0b10-4564-804e-f40f45f79d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt" Jul 17 18:22:29.292527 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:29.292506 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/584c95b4-0b10-4564-804e-f40f45f79d89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt\" (UID: \"584c95b4-0b10-4564-804e-f40f45f79d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt" Jul 17 18:22:29.429265 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:29.429168 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt" Jul 17 18:22:29.567892 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:29.567865 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt"] Jul 17 18:22:29.569880 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:22:29.569857 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584c95b4_0b10_4564_804e_f40f45f79d89.slice/crio-11aa0f64c26d9c90664aeebba9026619c8d9797fbd086c1365833a14eeefa33f WatchSource:0}: Error finding container 11aa0f64c26d9c90664aeebba9026619c8d9797fbd086c1365833a14eeefa33f: Status 404 returned error can't find the container with id 11aa0f64c26d9c90664aeebba9026619c8d9797fbd086c1365833a14eeefa33f Jul 17 18:22:30.337871 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:30.337836 2574 generic.go:356] "Generic (PLEG): container finished" podID="584c95b4-0b10-4564-804e-f40f45f79d89" containerID="a4edad66b936e2bd5ff72cb8e68286275cc8238883ed9aedc2ee79f52604bfd9" exitCode=7 Jul 17 18:22:30.338250 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:30.337915 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt" event={"ID":"584c95b4-0b10-4564-804e-f40f45f79d89","Type":"ContainerDied","Data":"a4edad66b936e2bd5ff72cb8e68286275cc8238883ed9aedc2ee79f52604bfd9"} Jul 17 18:22:30.338250 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:30.337964 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt" event={"ID":"584c95b4-0b10-4564-804e-f40f45f79d89","Type":"ContainerStarted","Data":"11aa0f64c26d9c90664aeebba9026619c8d9797fbd086c1365833a14eeefa33f"} Jul 17 18:22:30.338250 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:30.338142 2574 scope.go:117] "RemoveContainer" containerID="a4edad66b936e2bd5ff72cb8e68286275cc8238883ed9aedc2ee79f52604bfd9" Jul 17 18:22:30.687693 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:30.687657 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2"] Jul 17 18:22:30.688118 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:30.688105 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66e96f9-8859-4152-9e30-ecaa14b45151" containerName="revoke-keys" Jul 17 18:22:30.691110 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:30.691094 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2" Jul 17 18:22:30.697763 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:30.697734 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2"] Jul 17 18:22:30.805716 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:30.805673 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c8e321b-c6c2-4246-8a42-da77cad9bb15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2\" (UID: \"1c8e321b-c6c2-4246-8a42-da77cad9bb15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2" Jul 17 18:22:30.907152 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:30.907057 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c8e321b-c6c2-4246-8a42-da77cad9bb15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2\" (UID: \"1c8e321b-c6c2-4246-8a42-da77cad9bb15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2" Jul 17 18:22:30.907749 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:30.907730 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c8e321b-c6c2-4246-8a42-da77cad9bb15-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2\" (UID: \"1c8e321b-c6c2-4246-8a42-da77cad9bb15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2" Jul 17 18:22:31.002878 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:31.002843 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2" Jul 17 18:22:31.138807 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:31.138780 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2"] Jul 17 18:22:31.141092 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:22:31.141062 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8e321b_c6c2_4246_8a42_da77cad9bb15.slice/crio-9c786e8ae42327ac9c267ac27abcc03632b242c267025f448a1bf86e4989006b WatchSource:0}: Error finding container 9c786e8ae42327ac9c267ac27abcc03632b242c267025f448a1bf86e4989006b: Status 404 returned error can't find the container with id 9c786e8ae42327ac9c267ac27abcc03632b242c267025f448a1bf86e4989006b Jul 17 18:22:31.343975 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:31.343935 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2" event={"ID":"1c8e321b-c6c2-4246-8a42-da77cad9bb15","Type":"ContainerStarted","Data":"9c786e8ae42327ac9c267ac27abcc03632b242c267025f448a1bf86e4989006b"} Jul 17 18:22:31.345901 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:31.345875 2574 generic.go:356] "Generic (PLEG): container finished" podID="584c95b4-0b10-4564-804e-f40f45f79d89" containerID="c5ebebb27d31c30c059a57dae364d808cc282cad877c9191f2c76ad7ba48a55a" exitCode=7 Jul 17 18:22:31.346051 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:31.345960 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt" event={"ID":"584c95b4-0b10-4564-804e-f40f45f79d89","Type":"ContainerDied","Data":"c5ebebb27d31c30c059a57dae364d808cc282cad877c9191f2c76ad7ba48a55a"} Jul 17 18:22:31.346051 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:31.346023 2574 scope.go:117] "RemoveContainer" containerID="a4edad66b936e2bd5ff72cb8e68286275cc8238883ed9aedc2ee79f52604bfd9" Jul 17 18:22:31.346167 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:31.346146 2574 scope.go:117] "RemoveContainer" containerID="c5ebebb27d31c30c059a57dae364d808cc282cad877c9191f2c76ad7ba48a55a" Jul 17 18:22:31.346427 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:22:31.346405 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt_odh-ai-gateway-infra(584c95b4-0b10-4564-804e-f40f45f79d89)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt" podUID="584c95b4-0b10-4564-804e-f40f45f79d89" Jul 17 18:22:32.352408 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:32.352342 2574 generic.go:356] "Generic (PLEG): container finished" podID="1c8e321b-c6c2-4246-8a42-da77cad9bb15" containerID="a2c7c984c448ec3ecb8825c61fa2999f003b5b03a055ca6a7d1c88308f170a64" exitCode=6 Jul 17 18:22:32.352888 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:32.352426 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2" event={"ID":"1c8e321b-c6c2-4246-8a42-da77cad9bb15","Type":"ContainerDied","Data":"a2c7c984c448ec3ecb8825c61fa2999f003b5b03a055ca6a7d1c88308f170a64"} Jul 17 18:22:32.352888 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:32.352626 2574 scope.go:117] "RemoveContainer" containerID="a2c7c984c448ec3ecb8825c61fa2999f003b5b03a055ca6a7d1c88308f170a64" Jul 17 18:22:32.360557 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:32.354655 2574 scope.go:117] "RemoveContainer" containerID="c5ebebb27d31c30c059a57dae364d808cc282cad877c9191f2c76ad7ba48a55a" Jul 17 18:22:32.360557 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:22:32.355036 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt_odh-ai-gateway-infra(584c95b4-0b10-4564-804e-f40f45f79d89)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt" podUID="584c95b4-0b10-4564-804e-f40f45f79d89" Jul 17 18:22:33.361819 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:33.361721 2574 generic.go:356] "Generic (PLEG): container finished" podID="1c8e321b-c6c2-4246-8a42-da77cad9bb15" containerID="a2d71ade010d6bcc7615b79cbecf7f8b0f078d0f5a3146e2d19142a4e274acdf" exitCode=6 Jul 17 18:22:33.361819 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:33.361807 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2" event={"ID":"1c8e321b-c6c2-4246-8a42-da77cad9bb15","Type":"ContainerDied","Data":"a2d71ade010d6bcc7615b79cbecf7f8b0f078d0f5a3146e2d19142a4e274acdf"} Jul 17 18:22:33.362454 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:33.361869 2574 scope.go:117] "RemoveContainer" containerID="a2c7c984c448ec3ecb8825c61fa2999f003b5b03a055ca6a7d1c88308f170a64" Jul 17 18:22:33.362454 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:33.362139 2574 scope.go:117] "RemoveContainer" containerID="a2d71ade010d6bcc7615b79cbecf7f8b0f078d0f5a3146e2d19142a4e274acdf" Jul 17 18:22:33.362454 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:22:33.362347 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2_odh-ai-gateway-infra(1c8e321b-c6c2-4246-8a42-da77cad9bb15)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2" podUID="1c8e321b-c6c2-4246-8a42-da77cad9bb15" Jul 17 18:22:33.916446 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:33.916409 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:22:34.369424 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:34.369388 2574 scope.go:117] "RemoveContainer" containerID="a2d71ade010d6bcc7615b79cbecf7f8b0f078d0f5a3146e2d19142a4e274acdf" Jul 17 18:22:34.369806 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:22:34.369605 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2_odh-ai-gateway-infra(1c8e321b-c6c2-4246-8a42-da77cad9bb15)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2" podUID="1c8e321b-c6c2-4246-8a42-da77cad9bb15" Jul 17 18:22:44.744252 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:44.744215 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:22:44.979917 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:44.979886 2574 scope.go:117] "RemoveContainer" containerID="c5ebebb27d31c30c059a57dae364d808cc282cad877c9191f2c76ad7ba48a55a" Jul 17 18:22:44.980135 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:44.980004 2574 scope.go:117] "RemoveContainer" containerID="a2d71ade010d6bcc7615b79cbecf7f8b0f078d0f5a3146e2d19142a4e274acdf" Jul 17 18:22:45.431036 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:45.430997 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2" event={"ID":"1c8e321b-c6c2-4246-8a42-da77cad9bb15","Type":"ContainerStarted","Data":"2055a5175762bef5330d83a2ad1da8a9856974f5b42956f11c57554ab0f45fad"} Jul 17 18:22:45.432764 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:45.432741 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt" event={"ID":"584c95b4-0b10-4564-804e-f40f45f79d89","Type":"ContainerStarted","Data":"85dafa2377e49be18d07f3513303da3fd7e8479ef41fc106c1e22f9f6bd3fcb7"} Jul 17 18:22:45.447687 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:45.447634 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2" podStartSLOduration=15.44761832 podStartE2EDuration="15.44761832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:22:45.446014263 +0000 UTC m=+1977.033966964" watchObservedRunningTime="2026-07-17 18:22:45.44761832 +0000 UTC m=+1977.035571010" Jul 17 18:22:45.870103 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:22:45.870049 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584c95b4_0b10_4564_804e_f40f45f79d89.slice/crio-conmon-85dafa2377e49be18d07f3513303da3fd7e8479ef41fc106c1e22f9f6bd3fcb7.scope\": RecentStats: unable to find data in memory cache]" Jul 17 18:22:46.440193 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:46.440157 2574 generic.go:356] "Generic (PLEG): container finished" podID="1c8e321b-c6c2-4246-8a42-da77cad9bb15" containerID="2055a5175762bef5330d83a2ad1da8a9856974f5b42956f11c57554ab0f45fad" exitCode=6 Jul 17 18:22:46.440406 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:46.440230 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2" event={"ID":"1c8e321b-c6c2-4246-8a42-da77cad9bb15","Type":"ContainerDied","Data":"2055a5175762bef5330d83a2ad1da8a9856974f5b42956f11c57554ab0f45fad"} Jul 17 18:22:46.440406 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:46.440303 2574 scope.go:117] "RemoveContainer" containerID="a2d71ade010d6bcc7615b79cbecf7f8b0f078d0f5a3146e2d19142a4e274acdf" Jul 17 18:22:46.440680 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:46.440649 2574 scope.go:117] "RemoveContainer" containerID="2055a5175762bef5330d83a2ad1da8a9856974f5b42956f11c57554ab0f45fad" Jul 17 18:22:46.440930 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:22:46.440902 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2_odh-ai-gateway-infra(1c8e321b-c6c2-4246-8a42-da77cad9bb15)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2" podUID="1c8e321b-c6c2-4246-8a42-da77cad9bb15" Jul 17 18:22:46.442262 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:46.442243 2574 generic.go:356] "Generic (PLEG): container finished" podID="584c95b4-0b10-4564-804e-f40f45f79d89" containerID="85dafa2377e49be18d07f3513303da3fd7e8479ef41fc106c1e22f9f6bd3fcb7" exitCode=7 Jul 17 18:22:46.442374 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:46.442315 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt" event={"ID":"584c95b4-0b10-4564-804e-f40f45f79d89","Type":"ContainerDied","Data":"85dafa2377e49be18d07f3513303da3fd7e8479ef41fc106c1e22f9f6bd3fcb7"} Jul 17 18:22:46.442626 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:46.442613 2574 scope.go:117] "RemoveContainer" containerID="85dafa2377e49be18d07f3513303da3fd7e8479ef41fc106c1e22f9f6bd3fcb7" Jul 17 18:22:46.442796 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:22:46.442781 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt_odh-ai-gateway-infra(584c95b4-0b10-4564-804e-f40f45f79d89)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt" podUID="584c95b4-0b10-4564-804e-f40f45f79d89" Jul 17 18:22:46.452218 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:46.452202 2574 scope.go:117] "RemoveContainer" containerID="c5ebebb27d31c30c059a57dae364d808cc282cad877c9191f2c76ad7ba48a55a" Jul 17 18:22:46.455818 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:46.455776 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt" podStartSLOduration=17.455761547 podStartE2EDuration="17.455761547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:22:45.462422855 +0000 UTC m=+1977.050375545" watchObservedRunningTime="2026-07-17 18:22:46.455761547 +0000 UTC m=+1978.043714237" Jul 17 18:22:46.455974 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:46.455961 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2"] Jul 17 18:22:46.472695 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:46.472667 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt"] Jul 17 18:22:47.588978 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:47.588951 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt" Jul 17 18:22:47.623252 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:47.623228 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2" Jul 17 18:22:47.660894 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:47.660860 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/584c95b4-0b10-4564-804e-f40f45f79d89-maas-api-service-ca\") pod \"584c95b4-0b10-4564-804e-f40f45f79d89\" (UID: \"584c95b4-0b10-4564-804e-f40f45f79d89\") " Jul 17 18:22:47.661075 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:47.660940 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c8e321b-c6c2-4246-8a42-da77cad9bb15-maas-api-service-ca\") pod \"1c8e321b-c6c2-4246-8a42-da77cad9bb15\" (UID: \"1c8e321b-c6c2-4246-8a42-da77cad9bb15\") " Jul 17 18:22:47.661333 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:47.661308 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/584c95b4-0b10-4564-804e-f40f45f79d89-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "584c95b4-0b10-4564-804e-f40f45f79d89" (UID: "584c95b4-0b10-4564-804e-f40f45f79d89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:22:47.661428 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:47.661327 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c8e321b-c6c2-4246-8a42-da77cad9bb15-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1c8e321b-c6c2-4246-8a42-da77cad9bb15" (UID: "1c8e321b-c6c2-4246-8a42-da77cad9bb15"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:22:47.762559 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:47.762523 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c8e321b-c6c2-4246-8a42-da77cad9bb15-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:22:47.762559 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:47.762555 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/584c95b4-0b10-4564-804e-f40f45f79d89-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:22:48.458003 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:48.457973 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2" Jul 17 18:22:48.458215 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:48.457972 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2" event={"ID":"1c8e321b-c6c2-4246-8a42-da77cad9bb15","Type":"ContainerDied","Data":"9c786e8ae42327ac9c267ac27abcc03632b242c267025f448a1bf86e4989006b"} Jul 17 18:22:48.458215 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:48.458106 2574 scope.go:117] "RemoveContainer" containerID="2055a5175762bef5330d83a2ad1da8a9856974f5b42956f11c57554ab0f45fad" Jul 17 18:22:48.459487 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:48.459415 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt" Jul 17 18:22:48.459487 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:48.459430 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt" event={"ID":"584c95b4-0b10-4564-804e-f40f45f79d89","Type":"ContainerDied","Data":"11aa0f64c26d9c90664aeebba9026619c8d9797fbd086c1365833a14eeefa33f"} Jul 17 18:22:48.467437 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:48.467220 2574 scope.go:117] "RemoveContainer" containerID="85dafa2377e49be18d07f3513303da3fd7e8479ef41fc106c1e22f9f6bd3fcb7" Jul 17 18:22:48.487373 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:48.487320 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt"] Jul 17 18:22:48.493733 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:48.493700 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-prmzt"] Jul 17 18:22:48.504299 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:48.504263 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2"] Jul 17 18:22:48.507779 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:48.507755 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-9q5z2"] Jul 17 18:22:48.985749 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:48.985713 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c8e321b-c6c2-4246-8a42-da77cad9bb15" path="/var/lib/kubelet/pods/1c8e321b-c6c2-4246-8a42-da77cad9bb15/volumes" Jul 17 18:22:48.986116 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:48.986049 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="584c95b4-0b10-4564-804e-f40f45f79d89" path="/var/lib/kubelet/pods/584c95b4-0b10-4564-804e-f40f45f79d89/volumes" Jul 17 18:22:52.912731 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:52.912694 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n"] Jul 17 18:22:52.913154 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:52.913093 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="584c95b4-0b10-4564-804e-f40f45f79d89" containerName="revoke-keys" Jul 17 18:22:52.913154 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:52.913105 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="584c95b4-0b10-4564-804e-f40f45f79d89" containerName="revoke-keys" Jul 17 18:22:52.913154 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:52.913120 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c8e321b-c6c2-4246-8a42-da77cad9bb15" containerName="revoke-keys" Jul 17 18:22:52.913154 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:52.913126 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c8e321b-c6c2-4246-8a42-da77cad9bb15" containerName="revoke-keys" Jul 17 18:22:52.913154 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:52.913136 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c8e321b-c6c2-4246-8a42-da77cad9bb15" containerName="revoke-keys" Jul 17 18:22:52.913154 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:52.913142 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c8e321b-c6c2-4246-8a42-da77cad9bb15" containerName="revoke-keys" Jul 17 18:22:52.913154 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:52.913148 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c8e321b-c6c2-4246-8a42-da77cad9bb15" containerName="revoke-keys" Jul 17 18:22:52.913154 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:52.913154 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c8e321b-c6c2-4246-8a42-da77cad9bb15" containerName="revoke-keys" Jul 17 18:22:52.913495 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:52.913175 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="584c95b4-0b10-4564-804e-f40f45f79d89" containerName="revoke-keys" Jul 17 18:22:52.913495 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:52.913180 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="584c95b4-0b10-4564-804e-f40f45f79d89" containerName="revoke-keys" Jul 17 18:22:52.913495 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:52.913233 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c8e321b-c6c2-4246-8a42-da77cad9bb15" containerName="revoke-keys" Jul 17 18:22:52.913495 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:52.913241 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c8e321b-c6c2-4246-8a42-da77cad9bb15" containerName="revoke-keys" Jul 17 18:22:52.913495 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:52.913250 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="584c95b4-0b10-4564-804e-f40f45f79d89" containerName="revoke-keys" Jul 17 18:22:52.913495 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:52.913255 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="584c95b4-0b10-4564-804e-f40f45f79d89" containerName="revoke-keys" Jul 17 18:22:52.913495 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:52.913260 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="584c95b4-0b10-4564-804e-f40f45f79d89" containerName="revoke-keys" Jul 17 18:22:52.916336 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:52.916318 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n" Jul 17 18:22:52.918798 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:52.918765 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:22:52.918998 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:52.918812 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dpsm\"" Jul 17 18:22:52.922988 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:52.922960 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n"] Jul 17 18:22:53.012790 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:53.012749 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2570e52a-9bdd-4ecc-82fe-dd053d285df7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n\" (UID: \"2570e52a-9bdd-4ecc-82fe-dd053d285df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n" Jul 17 18:22:53.113921 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:53.113879 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2570e52a-9bdd-4ecc-82fe-dd053d285df7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n\" (UID: \"2570e52a-9bdd-4ecc-82fe-dd053d285df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n" Jul 17 18:22:53.114526 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:53.114506 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2570e52a-9bdd-4ecc-82fe-dd053d285df7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n\" (UID: \"2570e52a-9bdd-4ecc-82fe-dd053d285df7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n" Jul 17 18:22:53.228861 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:53.228826 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n" Jul 17 18:22:53.366044 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:53.366017 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n"] Jul 17 18:22:53.368425 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:22:53.368397 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2570e52a_9bdd_4ecc_82fe_dd053d285df7.slice/crio-a14fc740c7aa922d0bd06b90ed42e622411beb120e36f1790c19c1bffe1558dd WatchSource:0}: Error finding container a14fc740c7aa922d0bd06b90ed42e622411beb120e36f1790c19c1bffe1558dd: Status 404 returned error can't find the container with id a14fc740c7aa922d0bd06b90ed42e622411beb120e36f1790c19c1bffe1558dd Jul 17 18:22:53.486771 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:53.486680 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n" event={"ID":"2570e52a-9bdd-4ecc-82fe-dd053d285df7","Type":"ContainerStarted","Data":"a14fc740c7aa922d0bd06b90ed42e622411beb120e36f1790c19c1bffe1558dd"} Jul 17 18:22:54.494208 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:54.494166 2574 generic.go:356] "Generic (PLEG): container finished" podID="2570e52a-9bdd-4ecc-82fe-dd053d285df7" containerID="1615eeeed8246a4c9e058f358c8b465964bc7f522a38ab49f43f488761a1f020" exitCode=6 Jul 17 18:22:54.494632 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:54.494258 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n" event={"ID":"2570e52a-9bdd-4ecc-82fe-dd053d285df7","Type":"ContainerDied","Data":"1615eeeed8246a4c9e058f358c8b465964bc7f522a38ab49f43f488761a1f020"} Jul 17 18:22:54.494632 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:54.494434 2574 scope.go:117] "RemoveContainer" containerID="1615eeeed8246a4c9e058f358c8b465964bc7f522a38ab49f43f488761a1f020" Jul 17 18:22:55.500976 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:55.500942 2574 generic.go:356] "Generic (PLEG): container finished" podID="2570e52a-9bdd-4ecc-82fe-dd053d285df7" containerID="0dc815cb8a1748b7c17a6ec6141577f487863e9c2c71537055d984e0ef831d86" exitCode=6 Jul 17 18:22:55.501443 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:55.501013 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n" event={"ID":"2570e52a-9bdd-4ecc-82fe-dd053d285df7","Type":"ContainerDied","Data":"0dc815cb8a1748b7c17a6ec6141577f487863e9c2c71537055d984e0ef831d86"} Jul 17 18:22:55.501443 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:55.501065 2574 scope.go:117] "RemoveContainer" containerID="1615eeeed8246a4c9e058f358c8b465964bc7f522a38ab49f43f488761a1f020" Jul 17 18:22:55.501443 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:55.501315 2574 scope.go:117] "RemoveContainer" containerID="0dc815cb8a1748b7c17a6ec6141577f487863e9c2c71537055d984e0ef831d86" Jul 17 18:22:55.501614 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:22:55.501589 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n_odh-ai-gateway-infra(2570e52a-9bdd-4ecc-82fe-dd053d285df7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n" podUID="2570e52a-9bdd-4ecc-82fe-dd053d285df7" Jul 17 18:22:56.507654 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:22:56.507625 2574 scope.go:117] "RemoveContainer" containerID="0dc815cb8a1748b7c17a6ec6141577f487863e9c2c71537055d984e0ef831d86" Jul 17 18:22:56.508033 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:22:56.507839 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n_odh-ai-gateway-infra(2570e52a-9bdd-4ecc-82fe-dd053d285df7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n" podUID="2570e52a-9bdd-4ecc-82fe-dd053d285df7" Jul 17 18:23:00.327011 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:00.326973 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:23:00.429748 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:00.429709 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:23:09.108940 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:09.108896 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:23:09.205830 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:09.205794 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:23:10.979541 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:10.979509 2574 scope.go:117] "RemoveContainer" containerID="0dc815cb8a1748b7c17a6ec6141577f487863e9c2c71537055d984e0ef831d86" Jul 17 18:23:11.582919 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:11.582818 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n" event={"ID":"2570e52a-9bdd-4ecc-82fe-dd053d285df7","Type":"ContainerStarted","Data":"51d3c657af129ddbf942e3ff9cb3e1f38b2cd305d7b7f3ba97ded3089a8aa21c"} Jul 17 18:23:11.599012 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:11.598952 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n" podStartSLOduration=19.598935269000002 podStartE2EDuration="19.598935269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:23:11.59778611 +0000 UTC m=+2003.185738801" watchObservedRunningTime="2026-07-17 18:23:11.598935269 +0000 UTC m=+2003.186887959" Jul 17 18:23:12.590198 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:12.590161 2574 generic.go:356] "Generic (PLEG): container finished" podID="2570e52a-9bdd-4ecc-82fe-dd053d285df7" containerID="51d3c657af129ddbf942e3ff9cb3e1f38b2cd305d7b7f3ba97ded3089a8aa21c" exitCode=6 Jul 17 18:23:12.590652 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:12.590207 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n" event={"ID":"2570e52a-9bdd-4ecc-82fe-dd053d285df7","Type":"ContainerDied","Data":"51d3c657af129ddbf942e3ff9cb3e1f38b2cd305d7b7f3ba97ded3089a8aa21c"} Jul 17 18:23:12.590652 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:12.590256 2574 scope.go:117] "RemoveContainer" containerID="0dc815cb8a1748b7c17a6ec6141577f487863e9c2c71537055d984e0ef831d86" Jul 17 18:23:12.590652 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:12.590604 2574 scope.go:117] "RemoveContainer" containerID="51d3c657af129ddbf942e3ff9cb3e1f38b2cd305d7b7f3ba97ded3089a8aa21c" Jul 17 18:23:12.590906 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:23:12.590886 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n_odh-ai-gateway-infra(2570e52a-9bdd-4ecc-82fe-dd053d285df7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n" podUID="2570e52a-9bdd-4ecc-82fe-dd053d285df7" Jul 17 18:23:12.614795 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:12.614281 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n"] Jul 17 18:23:13.729965 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:13.729940 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n" Jul 17 18:23:13.816888 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:13.816850 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2570e52a-9bdd-4ecc-82fe-dd053d285df7-maas-api-service-ca\") pod \"2570e52a-9bdd-4ecc-82fe-dd053d285df7\" (UID: \"2570e52a-9bdd-4ecc-82fe-dd053d285df7\") " Jul 17 18:23:13.817258 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:13.817229 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2570e52a-9bdd-4ecc-82fe-dd053d285df7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2570e52a-9bdd-4ecc-82fe-dd053d285df7" (UID: "2570e52a-9bdd-4ecc-82fe-dd053d285df7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:23:13.918459 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:13.918335 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2570e52a-9bdd-4ecc-82fe-dd053d285df7-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:23:14.602750 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:14.602708 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n" event={"ID":"2570e52a-9bdd-4ecc-82fe-dd053d285df7","Type":"ContainerDied","Data":"a14fc740c7aa922d0bd06b90ed42e622411beb120e36f1790c19c1bffe1558dd"} Jul 17 18:23:14.602750 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:14.602731 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n" Jul 17 18:23:14.603020 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:14.602782 2574 scope.go:117] "RemoveContainer" containerID="51d3c657af129ddbf942e3ff9cb3e1f38b2cd305d7b7f3ba97ded3089a8aa21c" Jul 17 18:23:14.627696 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:14.627664 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n"] Jul 17 18:23:14.633504 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:14.633469 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-xl56n"] Jul 17 18:23:14.985571 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:14.985546 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2570e52a-9bdd-4ecc-82fe-dd053d285df7" path="/var/lib/kubelet/pods/2570e52a-9bdd-4ecc-82fe-dd053d285df7/volumes" Jul 17 18:23:19.147991 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:19.147955 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d"] Jul 17 18:23:19.148498 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:19.148341 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="584c95b4-0b10-4564-804e-f40f45f79d89" containerName="revoke-keys" Jul 17 18:23:19.148498 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:19.148366 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="584c95b4-0b10-4564-804e-f40f45f79d89" containerName="revoke-keys" Jul 17 18:23:19.148498 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:19.148388 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2570e52a-9bdd-4ecc-82fe-dd053d285df7" containerName="revoke-keys" Jul 17 18:23:19.148498 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:19.148397 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2570e52a-9bdd-4ecc-82fe-dd053d285df7" containerName="revoke-keys" Jul 17 18:23:19.148498 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:19.148417 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2570e52a-9bdd-4ecc-82fe-dd053d285df7" containerName="revoke-keys" Jul 17 18:23:19.148498 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:19.148425 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2570e52a-9bdd-4ecc-82fe-dd053d285df7" containerName="revoke-keys" Jul 17 18:23:19.148498 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:19.148435 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2570e52a-9bdd-4ecc-82fe-dd053d285df7" containerName="revoke-keys" Jul 17 18:23:19.148498 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:19.148442 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2570e52a-9bdd-4ecc-82fe-dd053d285df7" containerName="revoke-keys" Jul 17 18:23:19.148837 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:19.148528 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2570e52a-9bdd-4ecc-82fe-dd053d285df7" containerName="revoke-keys" Jul 17 18:23:19.148837 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:19.148539 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2570e52a-9bdd-4ecc-82fe-dd053d285df7" containerName="revoke-keys" Jul 17 18:23:19.148837 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:19.148548 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c8e321b-c6c2-4246-8a42-da77cad9bb15" containerName="revoke-keys" Jul 17 18:23:19.152829 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:19.152808 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d" Jul 17 18:23:19.156691 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:19.156668 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dpsm\"" Jul 17 18:23:19.156691 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:19.156678 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:23:19.158169 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:19.158146 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d"] Jul 17 18:23:19.272685 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:19.272643 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c719a86c-9968-4674-8f89-f0ae088b8505-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d\" (UID: \"c719a86c-9968-4674-8f89-f0ae088b8505\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d" Jul 17 18:23:19.373335 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:19.373295 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c719a86c-9968-4674-8f89-f0ae088b8505-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d\" (UID: \"c719a86c-9968-4674-8f89-f0ae088b8505\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d" Jul 17 18:23:19.373954 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:19.373932 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c719a86c-9968-4674-8f89-f0ae088b8505-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d\" (UID: \"c719a86c-9968-4674-8f89-f0ae088b8505\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d" Jul 17 18:23:19.472961 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:19.472928 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d" Jul 17 18:23:19.616300 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:19.616269 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d"] Jul 17 18:23:19.618093 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:23:19.618068 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc719a86c_9968_4674_8f89_f0ae088b8505.slice/crio-bd9bc815f9234ee3d6cd29d8abf34299ac0427f2aae29ea93e79311559bd6a70 WatchSource:0}: Error finding container bd9bc815f9234ee3d6cd29d8abf34299ac0427f2aae29ea93e79311559bd6a70: Status 404 returned error can't find the container with id bd9bc815f9234ee3d6cd29d8abf34299ac0427f2aae29ea93e79311559bd6a70 Jul 17 18:23:19.630933 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:19.630901 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d" event={"ID":"c719a86c-9968-4674-8f89-f0ae088b8505","Type":"ContainerStarted","Data":"bd9bc815f9234ee3d6cd29d8abf34299ac0427f2aae29ea93e79311559bd6a70"} Jul 17 18:23:20.637913 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:20.637874 2574 generic.go:356] "Generic (PLEG): container finished" podID="c719a86c-9968-4674-8f89-f0ae088b8505" containerID="a7034f2b721489d1b9d12548ceb0a56c6686a51537259dee18d6696fe627f2ff" exitCode=7 Jul 17 18:23:20.638311 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:20.637970 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d" event={"ID":"c719a86c-9968-4674-8f89-f0ae088b8505","Type":"ContainerDied","Data":"a7034f2b721489d1b9d12548ceb0a56c6686a51537259dee18d6696fe627f2ff"} Jul 17 18:23:20.638311 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:20.638157 2574 scope.go:117] "RemoveContainer" containerID="a7034f2b721489d1b9d12548ceb0a56c6686a51537259dee18d6696fe627f2ff" Jul 17 18:23:20.725826 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:20.725793 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7"] Jul 17 18:23:20.726261 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:20.726249 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2570e52a-9bdd-4ecc-82fe-dd053d285df7" containerName="revoke-keys" Jul 17 18:23:20.730624 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:20.730607 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7" Jul 17 18:23:20.735761 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:20.735730 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7"] Jul 17 18:23:20.886214 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:20.886185 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abade44-0353-412d-8818-6ce22375a62f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7\" (UID: \"7abade44-0353-412d-8818-6ce22375a62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7" Jul 17 18:23:20.987573 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:20.987542 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abade44-0353-412d-8818-6ce22375a62f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7\" (UID: \"7abade44-0353-412d-8818-6ce22375a62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7" Jul 17 18:23:20.988096 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:20.988078 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abade44-0353-412d-8818-6ce22375a62f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7\" (UID: \"7abade44-0353-412d-8818-6ce22375a62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7" Jul 17 18:23:21.043582 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:21.043537 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7" Jul 17 18:23:21.178953 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:21.178917 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7"] Jul 17 18:23:21.182425 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:23:21.182394 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abade44_0353_412d_8818_6ce22375a62f.slice/crio-5accbf4a8bed2d4dbbc018c9e0b2c81f9deb16ac4c32828c8b035bbba8af7a78 WatchSource:0}: Error finding container 5accbf4a8bed2d4dbbc018c9e0b2c81f9deb16ac4c32828c8b035bbba8af7a78: Status 404 returned error can't find the container with id 5accbf4a8bed2d4dbbc018c9e0b2c81f9deb16ac4c32828c8b035bbba8af7a78 Jul 17 18:23:21.644770 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:21.644659 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7" event={"ID":"7abade44-0353-412d-8818-6ce22375a62f","Type":"ContainerStarted","Data":"3fa495269aa44afc464e64ab8814c6620f295d3980230d4e7671a68d4334f905"} Jul 17 18:23:21.644770 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:21.644721 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7" event={"ID":"7abade44-0353-412d-8818-6ce22375a62f","Type":"ContainerStarted","Data":"5accbf4a8bed2d4dbbc018c9e0b2c81f9deb16ac4c32828c8b035bbba8af7a78"} Jul 17 18:23:21.646570 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:21.646549 2574 generic.go:356] "Generic (PLEG): container finished" podID="c719a86c-9968-4674-8f89-f0ae088b8505" containerID="381a9029e1844ed4f64fcb48294b3e8a1c59af628373b7e45095720820846d6a" exitCode=7 Jul 17 18:23:21.646663 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:21.646600 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d" event={"ID":"c719a86c-9968-4674-8f89-f0ae088b8505","Type":"ContainerDied","Data":"381a9029e1844ed4f64fcb48294b3e8a1c59af628373b7e45095720820846d6a"} Jul 17 18:23:21.646663 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:21.646643 2574 scope.go:117] "RemoveContainer" containerID="a7034f2b721489d1b9d12548ceb0a56c6686a51537259dee18d6696fe627f2ff" Jul 17 18:23:21.646766 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:21.646749 2574 scope.go:117] "RemoveContainer" containerID="381a9029e1844ed4f64fcb48294b3e8a1c59af628373b7e45095720820846d6a" Jul 17 18:23:21.647013 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:23:21.646986 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d_odh-ai-gateway-infra(c719a86c-9968-4674-8f89-f0ae088b8505)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d" podUID="c719a86c-9968-4674-8f89-f0ae088b8505" Jul 17 18:23:21.662613 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:21.662566 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7" podStartSLOduration=1.662549512 podStartE2EDuration="1.662549512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:23:21.659609649 +0000 UTC m=+2013.247562341" watchObservedRunningTime="2026-07-17 18:23:21.662549512 +0000 UTC m=+2013.250502247" Jul 17 18:23:22.656301 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:22.656265 2574 generic.go:356] "Generic (PLEG): container finished" podID="7abade44-0353-412d-8818-6ce22375a62f" containerID="3fa495269aa44afc464e64ab8814c6620f295d3980230d4e7671a68d4334f905" exitCode=6 Jul 17 18:23:22.656825 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:22.656346 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7" event={"ID":"7abade44-0353-412d-8818-6ce22375a62f","Type":"ContainerDied","Data":"3fa495269aa44afc464e64ab8814c6620f295d3980230d4e7671a68d4334f905"} Jul 17 18:23:22.656825 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:22.656622 2574 scope.go:117] "RemoveContainer" containerID="3fa495269aa44afc464e64ab8814c6620f295d3980230d4e7671a68d4334f905" Jul 17 18:23:22.658187 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:22.658172 2574 scope.go:117] "RemoveContainer" containerID="381a9029e1844ed4f64fcb48294b3e8a1c59af628373b7e45095720820846d6a" Jul 17 18:23:22.658389 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:23:22.658349 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d_odh-ai-gateway-infra(c719a86c-9968-4674-8f89-f0ae088b8505)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d" podUID="c719a86c-9968-4674-8f89-f0ae088b8505" Jul 17 18:23:23.673618 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:23.673516 2574 generic.go:356] "Generic (PLEG): container finished" podID="7abade44-0353-412d-8818-6ce22375a62f" containerID="e1271134daa105d53f4c2cbf159004fac7438f72d8f229a2a9b6261cf98d6e66" exitCode=6 Jul 17 18:23:23.673618 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:23.673597 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7" event={"ID":"7abade44-0353-412d-8818-6ce22375a62f","Type":"ContainerDied","Data":"e1271134daa105d53f4c2cbf159004fac7438f72d8f229a2a9b6261cf98d6e66"} Jul 17 18:23:23.674105 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:23.673680 2574 scope.go:117] "RemoveContainer" containerID="3fa495269aa44afc464e64ab8814c6620f295d3980230d4e7671a68d4334f905" Jul 17 18:23:23.674105 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:23.673891 2574 scope.go:117] "RemoveContainer" containerID="e1271134daa105d53f4c2cbf159004fac7438f72d8f229a2a9b6261cf98d6e66" Jul 17 18:23:23.674192 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:23:23.674140 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7_odh-ai-gateway-infra(7abade44-0353-412d-8818-6ce22375a62f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7" podUID="7abade44-0353-412d-8818-6ce22375a62f" Jul 17 18:23:24.681468 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:24.681420 2574 scope.go:117] "RemoveContainer" containerID="e1271134daa105d53f4c2cbf159004fac7438f72d8f229a2a9b6261cf98d6e66" Jul 17 18:23:24.681936 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:23:24.681716 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7_odh-ai-gateway-infra(7abade44-0353-412d-8818-6ce22375a62f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7" podUID="7abade44-0353-412d-8818-6ce22375a62f" Jul 17 18:23:35.979773 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:35.979738 2574 scope.go:117] "RemoveContainer" containerID="e1271134daa105d53f4c2cbf159004fac7438f72d8f229a2a9b6261cf98d6e66" Jul 17 18:23:36.748072 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:36.748041 2574 generic.go:356] "Generic (PLEG): container finished" podID="7abade44-0353-412d-8818-6ce22375a62f" containerID="5548c1794598f6ae071e2b4e1a91bda00da08d11e98f31afeabc4a8562a4dfad" exitCode=6 Jul 17 18:23:36.748223 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:36.748121 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7" event={"ID":"7abade44-0353-412d-8818-6ce22375a62f","Type":"ContainerDied","Data":"5548c1794598f6ae071e2b4e1a91bda00da08d11e98f31afeabc4a8562a4dfad"} Jul 17 18:23:36.748223 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:36.748184 2574 scope.go:117] "RemoveContainer" containerID="e1271134daa105d53f4c2cbf159004fac7438f72d8f229a2a9b6261cf98d6e66" Jul 17 18:23:36.748560 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:36.748546 2574 scope.go:117] "RemoveContainer" containerID="5548c1794598f6ae071e2b4e1a91bda00da08d11e98f31afeabc4a8562a4dfad" Jul 17 18:23:36.748800 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:23:36.748780 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7_odh-ai-gateway-infra(7abade44-0353-412d-8818-6ce22375a62f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7" podUID="7abade44-0353-412d-8818-6ce22375a62f" Jul 17 18:23:37.773536 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:37.773497 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7"] Jul 17 18:23:37.918798 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:37.918770 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7" Jul 17 18:23:37.955583 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:37.955552 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abade44-0353-412d-8818-6ce22375a62f-maas-api-service-ca\") pod \"7abade44-0353-412d-8818-6ce22375a62f\" (UID: \"7abade44-0353-412d-8818-6ce22375a62f\") " Jul 17 18:23:37.955916 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:37.955892 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abade44-0353-412d-8818-6ce22375a62f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7abade44-0353-412d-8818-6ce22375a62f" (UID: "7abade44-0353-412d-8818-6ce22375a62f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:23:37.979921 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:37.979891 2574 scope.go:117] "RemoveContainer" containerID="381a9029e1844ed4f64fcb48294b3e8a1c59af628373b7e45095720820846d6a" Jul 17 18:23:38.056823 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:38.056724 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abade44-0353-412d-8818-6ce22375a62f-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:23:38.762134 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:38.762104 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7" Jul 17 18:23:38.762134 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:38.762117 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7" event={"ID":"7abade44-0353-412d-8818-6ce22375a62f","Type":"ContainerDied","Data":"5accbf4a8bed2d4dbbc018c9e0b2c81f9deb16ac4c32828c8b035bbba8af7a78"} Jul 17 18:23:38.762440 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:38.762191 2574 scope.go:117] "RemoveContainer" containerID="5548c1794598f6ae071e2b4e1a91bda00da08d11e98f31afeabc4a8562a4dfad" Jul 17 18:23:38.764334 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:38.764307 2574 generic.go:356] "Generic (PLEG): container finished" podID="c719a86c-9968-4674-8f89-f0ae088b8505" containerID="09a42e582e9061caf3ee9eac49249eaba0dfce4e3f16d1d42e5de9575f700017" exitCode=7 Jul 17 18:23:38.764453 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:38.764393 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d" event={"ID":"c719a86c-9968-4674-8f89-f0ae088b8505","Type":"ContainerDied","Data":"09a42e582e9061caf3ee9eac49249eaba0dfce4e3f16d1d42e5de9575f700017"} Jul 17 18:23:38.764690 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:38.764676 2574 scope.go:117] "RemoveContainer" containerID="09a42e582e9061caf3ee9eac49249eaba0dfce4e3f16d1d42e5de9575f700017" Jul 17 18:23:38.764869 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:23:38.764852 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d_odh-ai-gateway-infra(c719a86c-9968-4674-8f89-f0ae088b8505)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d" podUID="c719a86c-9968-4674-8f89-f0ae088b8505" Jul 17 18:23:38.772130 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:38.772113 2574 scope.go:117] "RemoveContainer" containerID="381a9029e1844ed4f64fcb48294b3e8a1c59af628373b7e45095720820846d6a" Jul 17 18:23:38.795484 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:38.795453 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7"] Jul 17 18:23:38.800717 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:38.800689 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-wcxp7"] Jul 17 18:23:38.985556 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:38.985517 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7abade44-0353-412d-8818-6ce22375a62f" path="/var/lib/kubelet/pods/7abade44-0353-412d-8818-6ce22375a62f/volumes" Jul 17 18:23:39.792504 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:39.792468 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d"] Jul 17 18:23:39.928983 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:39.928958 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d" Jul 17 18:23:39.975040 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:39.975001 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c719a86c-9968-4674-8f89-f0ae088b8505-maas-api-service-ca\") pod \"c719a86c-9968-4674-8f89-f0ae088b8505\" (UID: \"c719a86c-9968-4674-8f89-f0ae088b8505\") " Jul 17 18:23:39.975428 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:39.975409 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c719a86c-9968-4674-8f89-f0ae088b8505-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c719a86c-9968-4674-8f89-f0ae088b8505" (UID: "c719a86c-9968-4674-8f89-f0ae088b8505"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:23:40.076156 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:40.076064 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c719a86c-9968-4674-8f89-f0ae088b8505-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:23:40.778919 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:40.778890 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d" Jul 17 18:23:40.779101 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:40.778888 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d" event={"ID":"c719a86c-9968-4674-8f89-f0ae088b8505","Type":"ContainerDied","Data":"bd9bc815f9234ee3d6cd29d8abf34299ac0427f2aae29ea93e79311559bd6a70"} Jul 17 18:23:40.779101 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:40.779023 2574 scope.go:117] "RemoveContainer" containerID="09a42e582e9061caf3ee9eac49249eaba0dfce4e3f16d1d42e5de9575f700017" Jul 17 18:23:40.798773 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:40.798745 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d"] Jul 17 18:23:40.804869 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:40.804846 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-wd76d"] Jul 17 18:23:40.985269 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:40.985235 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c719a86c-9968-4674-8f89-f0ae088b8505" path="/var/lib/kubelet/pods/c719a86c-9968-4674-8f89-f0ae088b8505/volumes" Jul 17 18:23:45.637582 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:45.637540 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:23:52.949886 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:52.949849 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx"] Jul 17 18:23:52.950325 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:52.950271 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c719a86c-9968-4674-8f89-f0ae088b8505" containerName="revoke-keys" Jul 17 18:23:52.950325 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:52.950284 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c719a86c-9968-4674-8f89-f0ae088b8505" containerName="revoke-keys" Jul 17 18:23:52.950325 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:52.950292 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7abade44-0353-412d-8818-6ce22375a62f" containerName="revoke-keys" Jul 17 18:23:52.950325 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:52.950298 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7abade44-0353-412d-8818-6ce22375a62f" containerName="revoke-keys" Jul 17 18:23:52.950325 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:52.950306 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c719a86c-9968-4674-8f89-f0ae088b8505" containerName="revoke-keys" Jul 17 18:23:52.950325 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:52.950312 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c719a86c-9968-4674-8f89-f0ae088b8505" containerName="revoke-keys" Jul 17 18:23:52.950325 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:52.950318 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c719a86c-9968-4674-8f89-f0ae088b8505" containerName="revoke-keys" Jul 17 18:23:52.950325 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:52.950324 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c719a86c-9968-4674-8f89-f0ae088b8505" containerName="revoke-keys" Jul 17 18:23:52.950614 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:52.950334 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7abade44-0353-412d-8818-6ce22375a62f" containerName="revoke-keys" Jul 17 18:23:52.950614 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:52.950342 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7abade44-0353-412d-8818-6ce22375a62f" containerName="revoke-keys" Jul 17 18:23:52.950614 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:52.950348 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7abade44-0353-412d-8818-6ce22375a62f" containerName="revoke-keys" Jul 17 18:23:52.950614 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:52.950372 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7abade44-0353-412d-8818-6ce22375a62f" containerName="revoke-keys" Jul 17 18:23:52.950614 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:52.950438 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c719a86c-9968-4674-8f89-f0ae088b8505" containerName="revoke-keys" Jul 17 18:23:52.950614 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:52.950445 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7abade44-0353-412d-8818-6ce22375a62f" containerName="revoke-keys" Jul 17 18:23:52.950614 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:52.950454 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c719a86c-9968-4674-8f89-f0ae088b8505" containerName="revoke-keys" Jul 17 18:23:52.950614 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:52.950460 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7abade44-0353-412d-8818-6ce22375a62f" containerName="revoke-keys" Jul 17 18:23:52.950614 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:52.950465 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7abade44-0353-412d-8818-6ce22375a62f" containerName="revoke-keys" Jul 17 18:23:52.953552 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:52.953535 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx" Jul 17 18:23:52.955961 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:52.955934 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dpsm\"" Jul 17 18:23:52.956090 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:52.955963 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:23:52.959869 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:52.959843 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx"] Jul 17 18:23:53.103963 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:53.103922 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e8de1d-74a0-4b63-b91f-712e2343fe6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx\" (UID: \"b7e8de1d-74a0-4b63-b91f-712e2343fe6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx" Jul 17 18:23:53.205327 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:53.205234 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e8de1d-74a0-4b63-b91f-712e2343fe6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx\" (UID: \"b7e8de1d-74a0-4b63-b91f-712e2343fe6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx" Jul 17 18:23:53.205930 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:53.205909 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e8de1d-74a0-4b63-b91f-712e2343fe6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx\" (UID: \"b7e8de1d-74a0-4b63-b91f-712e2343fe6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx" Jul 17 18:23:53.265089 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:53.265052 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx" Jul 17 18:23:53.403508 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:53.403487 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx"] Jul 17 18:23:53.405979 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:23:53.405953 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e8de1d_74a0_4b63_b91f_712e2343fe6f.slice/crio-558a0915f1be2ee680b45ecf225489fee6d3121220561465721dd5357ef5aeb2 WatchSource:0}: Error finding container 558a0915f1be2ee680b45ecf225489fee6d3121220561465721dd5357ef5aeb2: Status 404 returned error can't find the container with id 558a0915f1be2ee680b45ecf225489fee6d3121220561465721dd5357ef5aeb2 Jul 17 18:23:53.848924 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:53.848824 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx" event={"ID":"b7e8de1d-74a0-4b63-b91f-712e2343fe6f","Type":"ContainerStarted","Data":"a209bead9fc073cf78b4ff0f3f9567f361ece8b03ecebf1206e69e71721bca20"} Jul 17 18:23:53.848924 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:53.848880 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx" event={"ID":"b7e8de1d-74a0-4b63-b91f-712e2343fe6f","Type":"ContainerStarted","Data":"558a0915f1be2ee680b45ecf225489fee6d3121220561465721dd5357ef5aeb2"} Jul 17 18:23:53.865785 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:53.865732 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx" podStartSLOduration=1.865715903 podStartE2EDuration="1.865715903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:23:53.864621929 +0000 UTC m=+2045.452574620" watchObservedRunningTime="2026-07-17 18:23:53.865715903 +0000 UTC m=+2045.453668592" Jul 17 18:23:54.860943 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:54.860898 2574 generic.go:356] "Generic (PLEG): container finished" podID="b7e8de1d-74a0-4b63-b91f-712e2343fe6f" containerID="a209bead9fc073cf78b4ff0f3f9567f361ece8b03ecebf1206e69e71721bca20" exitCode=6 Jul 17 18:23:54.860943 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:54.860946 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx" event={"ID":"b7e8de1d-74a0-4b63-b91f-712e2343fe6f","Type":"ContainerDied","Data":"a209bead9fc073cf78b4ff0f3f9567f361ece8b03ecebf1206e69e71721bca20"} Jul 17 18:23:54.861439 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:54.861186 2574 scope.go:117] "RemoveContainer" containerID="a209bead9fc073cf78b4ff0f3f9567f361ece8b03ecebf1206e69e71721bca20" Jul 17 18:23:55.868092 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:55.867999 2574 generic.go:356] "Generic (PLEG): container finished" podID="b7e8de1d-74a0-4b63-b91f-712e2343fe6f" containerID="f5feb9cc2a8a51f9778e483612dfba83b50b54d4029ef83add137883e6a63a71" exitCode=6 Jul 17 18:23:55.868092 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:55.868083 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx" event={"ID":"b7e8de1d-74a0-4b63-b91f-712e2343fe6f","Type":"ContainerDied","Data":"f5feb9cc2a8a51f9778e483612dfba83b50b54d4029ef83add137883e6a63a71"} Jul 17 18:23:55.868600 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:55.868142 2574 scope.go:117] "RemoveContainer" containerID="a209bead9fc073cf78b4ff0f3f9567f361ece8b03ecebf1206e69e71721bca20" Jul 17 18:23:55.868600 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:55.868347 2574 scope.go:117] "RemoveContainer" containerID="f5feb9cc2a8a51f9778e483612dfba83b50b54d4029ef83add137883e6a63a71" Jul 17 18:23:55.868600 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:23:55.868583 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx_odh-ai-gateway-infra(b7e8de1d-74a0-4b63-b91f-712e2343fe6f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx" podUID="b7e8de1d-74a0-4b63-b91f-712e2343fe6f" Jul 17 18:23:56.876222 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:23:56.876188 2574 scope.go:117] "RemoveContainer" containerID="f5feb9cc2a8a51f9778e483612dfba83b50b54d4029ef83add137883e6a63a71" Jul 17 18:23:56.876628 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:23:56.876412 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx_odh-ai-gateway-infra(b7e8de1d-74a0-4b63-b91f-712e2343fe6f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx" podUID="b7e8de1d-74a0-4b63-b91f-712e2343fe6f" Jul 17 18:24:08.982153 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:08.982124 2574 scope.go:117] "RemoveContainer" containerID="f5feb9cc2a8a51f9778e483612dfba83b50b54d4029ef83add137883e6a63a71" Jul 17 18:24:09.940920 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:09.940810 2574 generic.go:356] "Generic (PLEG): container finished" podID="b7e8de1d-74a0-4b63-b91f-712e2343fe6f" containerID="28d7b5f03a1a17c279fefc4a924d2d1482616c0329b3261c17367876988b23ba" exitCode=6 Jul 17 18:24:09.940920 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:09.940880 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx" event={"ID":"b7e8de1d-74a0-4b63-b91f-712e2343fe6f","Type":"ContainerDied","Data":"28d7b5f03a1a17c279fefc4a924d2d1482616c0329b3261c17367876988b23ba"} Jul 17 18:24:09.941134 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:09.940945 2574 scope.go:117] "RemoveContainer" containerID="f5feb9cc2a8a51f9778e483612dfba83b50b54d4029ef83add137883e6a63a71" Jul 17 18:24:09.941234 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:09.941218 2574 scope.go:117] "RemoveContainer" containerID="28d7b5f03a1a17c279fefc4a924d2d1482616c0329b3261c17367876988b23ba" Jul 17 18:24:09.941485 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:24:09.941465 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx_odh-ai-gateway-infra(b7e8de1d-74a0-4b63-b91f-712e2343fe6f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx" podUID="b7e8de1d-74a0-4b63-b91f-712e2343fe6f" Jul 17 18:24:10.762666 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:10.762629 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-89hft"] Jul 17 18:24:10.763103 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:10.763090 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c719a86c-9968-4674-8f89-f0ae088b8505" containerName="revoke-keys" Jul 17 18:24:10.767496 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:10.767478 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-89hft" Jul 17 18:24:10.773058 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:10.773032 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-89hft"] Jul 17 18:24:10.871864 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:10.871821 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd81081-065a-4e7e-af7d-d5937b216de6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-89hft\" (UID: \"bdd81081-065a-4e7e-af7d-d5937b216de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-89hft" Jul 17 18:24:10.966625 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:10.966585 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx"] Jul 17 18:24:10.972651 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:10.972612 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd81081-065a-4e7e-af7d-d5937b216de6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-89hft\" (UID: \"bdd81081-065a-4e7e-af7d-d5937b216de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-89hft" Jul 17 18:24:10.973531 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:10.973507 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd81081-065a-4e7e-af7d-d5937b216de6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-89hft\" (UID: \"bdd81081-065a-4e7e-af7d-d5937b216de6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-89hft" Jul 17 18:24:11.079498 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:11.079464 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-89hft" Jul 17 18:24:11.112945 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:11.112918 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx" Jul 17 18:24:11.175077 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:11.175039 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e8de1d-74a0-4b63-b91f-712e2343fe6f-maas-api-service-ca\") pod \"b7e8de1d-74a0-4b63-b91f-712e2343fe6f\" (UID: \"b7e8de1d-74a0-4b63-b91f-712e2343fe6f\") " Jul 17 18:24:11.175871 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:11.175821 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7e8de1d-74a0-4b63-b91f-712e2343fe6f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b7e8de1d-74a0-4b63-b91f-712e2343fe6f" (UID: "b7e8de1d-74a0-4b63-b91f-712e2343fe6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:24:11.225776 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:11.225752 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-89hft"] Jul 17 18:24:11.227435 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:24:11.227406 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd81081_065a_4e7e_af7d_d5937b216de6.slice/crio-4ecfbec90e2857a29225af382d399ab02920a74dc7599712293a31d71d490a2d WatchSource:0}: Error finding container 4ecfbec90e2857a29225af382d399ab02920a74dc7599712293a31d71d490a2d: Status 404 returned error can't find the container with id 4ecfbec90e2857a29225af382d399ab02920a74dc7599712293a31d71d490a2d Jul 17 18:24:11.276714 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:11.276633 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e8de1d-74a0-4b63-b91f-712e2343fe6f-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:24:11.953751 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:11.953727 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx" Jul 17 18:24:11.953751 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:11.953730 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx" event={"ID":"b7e8de1d-74a0-4b63-b91f-712e2343fe6f","Type":"ContainerDied","Data":"558a0915f1be2ee680b45ecf225489fee6d3121220561465721dd5357ef5aeb2"} Jul 17 18:24:11.954294 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:11.953804 2574 scope.go:117] "RemoveContainer" containerID="28d7b5f03a1a17c279fefc4a924d2d1482616c0329b3261c17367876988b23ba" Jul 17 18:24:11.955666 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:11.955640 2574 generic.go:356] "Generic (PLEG): container finished" podID="bdd81081-065a-4e7e-af7d-d5937b216de6" containerID="3959db38eb04fbf1699060b1e1a0e89a627ecde761b9346af3bd158abe9411d6" exitCode=6 Jul 17 18:24:11.955794 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:11.955717 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-89hft" event={"ID":"bdd81081-065a-4e7e-af7d-d5937b216de6","Type":"ContainerDied","Data":"3959db38eb04fbf1699060b1e1a0e89a627ecde761b9346af3bd158abe9411d6"} Jul 17 18:24:11.955794 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:11.955757 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-89hft" event={"ID":"bdd81081-065a-4e7e-af7d-d5937b216de6","Type":"ContainerStarted","Data":"4ecfbec90e2857a29225af382d399ab02920a74dc7599712293a31d71d490a2d"} Jul 17 18:24:11.955916 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:11.955904 2574 scope.go:117] "RemoveContainer" containerID="3959db38eb04fbf1699060b1e1a0e89a627ecde761b9346af3bd158abe9411d6" Jul 17 18:24:11.986228 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:11.986196 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx"] Jul 17 18:24:11.991641 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:11.991620 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wqwvx"] Jul 17 18:24:12.963759 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:12.963666 2574 generic.go:356] "Generic (PLEG): container finished" podID="bdd81081-065a-4e7e-af7d-d5937b216de6" containerID="23e78fb3c7faf5bd2ee7af1bf805fcf24f25320f765e53ddaf9cc28af1da12f0" exitCode=6 Jul 17 18:24:12.964169 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:12.963754 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-89hft" event={"ID":"bdd81081-065a-4e7e-af7d-d5937b216de6","Type":"ContainerDied","Data":"23e78fb3c7faf5bd2ee7af1bf805fcf24f25320f765e53ddaf9cc28af1da12f0"} Jul 17 18:24:12.964169 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:12.963853 2574 scope.go:117] "RemoveContainer" containerID="3959db38eb04fbf1699060b1e1a0e89a627ecde761b9346af3bd158abe9411d6" Jul 17 18:24:12.964169 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:12.963981 2574 scope.go:117] "RemoveContainer" containerID="23e78fb3c7faf5bd2ee7af1bf805fcf24f25320f765e53ddaf9cc28af1da12f0" Jul 17 18:24:12.964313 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:24:12.964269 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-89hft_odh-ai-gateway-infra(bdd81081-065a-4e7e-af7d-d5937b216de6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-89hft" podUID="bdd81081-065a-4e7e-af7d-d5937b216de6" Jul 17 18:24:12.985625 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:12.985585 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7e8de1d-74a0-4b63-b91f-712e2343fe6f" path="/var/lib/kubelet/pods/b7e8de1d-74a0-4b63-b91f-712e2343fe6f/volumes" Jul 17 18:24:13.970977 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:13.970949 2574 scope.go:117] "RemoveContainer" containerID="23e78fb3c7faf5bd2ee7af1bf805fcf24f25320f765e53ddaf9cc28af1da12f0" Jul 17 18:24:13.971461 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:24:13.971249 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-89hft_odh-ai-gateway-infra(bdd81081-065a-4e7e-af7d-d5937b216de6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-89hft" podUID="bdd81081-065a-4e7e-af7d-d5937b216de6" Jul 17 18:24:19.190473 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:19.190441 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4"] Jul 17 18:24:19.190897 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:19.190829 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7e8de1d-74a0-4b63-b91f-712e2343fe6f" containerName="revoke-keys" Jul 17 18:24:19.190897 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:19.190840 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7e8de1d-74a0-4b63-b91f-712e2343fe6f" containerName="revoke-keys" Jul 17 18:24:19.190897 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:19.190864 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7e8de1d-74a0-4b63-b91f-712e2343fe6f" containerName="revoke-keys" Jul 17 18:24:19.190897 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:19.190869 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7e8de1d-74a0-4b63-b91f-712e2343fe6f" containerName="revoke-keys" Jul 17 18:24:19.190897 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:19.190876 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7e8de1d-74a0-4b63-b91f-712e2343fe6f" containerName="revoke-keys" Jul 17 18:24:19.190897 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:19.190881 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7e8de1d-74a0-4b63-b91f-712e2343fe6f" containerName="revoke-keys" Jul 17 18:24:19.191119 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:19.190937 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7e8de1d-74a0-4b63-b91f-712e2343fe6f" containerName="revoke-keys" Jul 17 18:24:19.191119 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:19.190950 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7e8de1d-74a0-4b63-b91f-712e2343fe6f" containerName="revoke-keys" Jul 17 18:24:19.195574 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:19.195557 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4" Jul 17 18:24:19.200589 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:19.200557 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4"] Jul 17 18:24:19.254251 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:19.254208 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4\" (UID: \"ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4" Jul 17 18:24:19.354830 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:19.354796 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4\" (UID: \"ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4" Jul 17 18:24:19.355438 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:19.355418 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4\" (UID: \"ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4" Jul 17 18:24:19.507418 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:19.507383 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4" Jul 17 18:24:19.651920 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:19.651898 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4"] Jul 17 18:24:19.654640 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:24:19.654610 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0df30f_3cec_4c59_8ee6_e3ed11d1ba7e.slice/crio-dc6501083b8a86791ad368a5fd249ba1e56493174937a571cde03665fb9ee884 WatchSource:0}: Error finding container dc6501083b8a86791ad368a5fd249ba1e56493174937a571cde03665fb9ee884: Status 404 returned error can't find the container with id dc6501083b8a86791ad368a5fd249ba1e56493174937a571cde03665fb9ee884 Jul 17 18:24:20.001592 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:20.001561 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4" event={"ID":"ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e","Type":"ContainerStarted","Data":"d7497eaf8d6fae582123a6c003518dc3dcaefb5297899fb1685461fa90e0c2a5"} Jul 17 18:24:20.001783 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:20.001619 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4" event={"ID":"ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e","Type":"ContainerStarted","Data":"dc6501083b8a86791ad368a5fd249ba1e56493174937a571cde03665fb9ee884"} Jul 17 18:24:20.017049 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:20.016997 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4" podStartSLOduration=1.016980437 podStartE2EDuration="1.016980437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:24:20.015732601 +0000 UTC m=+2071.603685292" watchObservedRunningTime="2026-07-17 18:24:20.016980437 +0000 UTC m=+2071.604933172" Jul 17 18:24:21.007631 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:21.007590 2574 generic.go:356] "Generic (PLEG): container finished" podID="ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e" containerID="d7497eaf8d6fae582123a6c003518dc3dcaefb5297899fb1685461fa90e0c2a5" exitCode=7 Jul 17 18:24:21.008052 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:21.007682 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4" event={"ID":"ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e","Type":"ContainerDied","Data":"d7497eaf8d6fae582123a6c003518dc3dcaefb5297899fb1685461fa90e0c2a5"} Jul 17 18:24:21.008052 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:21.007875 2574 scope.go:117] "RemoveContainer" containerID="d7497eaf8d6fae582123a6c003518dc3dcaefb5297899fb1685461fa90e0c2a5" Jul 17 18:24:22.014908 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:22.014876 2574 generic.go:356] "Generic (PLEG): container finished" podID="ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e" containerID="c88356b4cb5d18fec822a4ef13dd235b732f96c6d4be0de0d016a478369625ab" exitCode=7 Jul 17 18:24:22.015302 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:22.014966 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4" event={"ID":"ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e","Type":"ContainerDied","Data":"c88356b4cb5d18fec822a4ef13dd235b732f96c6d4be0de0d016a478369625ab"} Jul 17 18:24:22.015302 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:22.015025 2574 scope.go:117] "RemoveContainer" containerID="d7497eaf8d6fae582123a6c003518dc3dcaefb5297899fb1685461fa90e0c2a5" Jul 17 18:24:22.015302 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:22.015165 2574 scope.go:117] "RemoveContainer" containerID="c88356b4cb5d18fec822a4ef13dd235b732f96c6d4be0de0d016a478369625ab" Jul 17 18:24:22.015521 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:24:22.015429 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4_odh-ai-gateway-infra(ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4" podUID="ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e" Jul 17 18:24:23.021983 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:23.021951 2574 scope.go:117] "RemoveContainer" containerID="c88356b4cb5d18fec822a4ef13dd235b732f96c6d4be0de0d016a478369625ab" Jul 17 18:24:23.022397 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:24:23.022165 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4_odh-ai-gateway-infra(ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4" podUID="ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e" Jul 17 18:24:23.979677 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:23.979642 2574 scope.go:117] "RemoveContainer" containerID="23e78fb3c7faf5bd2ee7af1bf805fcf24f25320f765e53ddaf9cc28af1da12f0" Jul 17 18:24:25.033404 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:25.033372 2574 generic.go:356] "Generic (PLEG): container finished" podID="bdd81081-065a-4e7e-af7d-d5937b216de6" containerID="0a9c97f4c07ff0af9040ab812144408910eee7310950f8e9d6ad08483f470b34" exitCode=6 Jul 17 18:24:25.033404 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:25.033393 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-89hft" event={"ID":"bdd81081-065a-4e7e-af7d-d5937b216de6","Type":"ContainerDied","Data":"0a9c97f4c07ff0af9040ab812144408910eee7310950f8e9d6ad08483f470b34"} Jul 17 18:24:25.033924 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:25.033472 2574 scope.go:117] "RemoveContainer" containerID="23e78fb3c7faf5bd2ee7af1bf805fcf24f25320f765e53ddaf9cc28af1da12f0" Jul 17 18:24:25.033924 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:25.033757 2574 scope.go:117] "RemoveContainer" containerID="0a9c97f4c07ff0af9040ab812144408910eee7310950f8e9d6ad08483f470b34" Jul 17 18:24:25.034002 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:24:25.033983 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-89hft_odh-ai-gateway-infra(bdd81081-065a-4e7e-af7d-d5937b216de6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-89hft" podUID="bdd81081-065a-4e7e-af7d-d5937b216de6" Jul 17 18:24:26.058710 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:26.058674 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-89hft"] Jul 17 18:24:26.196764 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:26.196731 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-89hft" Jul 17 18:24:26.323537 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:26.323440 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd81081-065a-4e7e-af7d-d5937b216de6-maas-api-service-ca\") pod \"bdd81081-065a-4e7e-af7d-d5937b216de6\" (UID: \"bdd81081-065a-4e7e-af7d-d5937b216de6\") " Jul 17 18:24:26.323815 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:26.323790 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd81081-065a-4e7e-af7d-d5937b216de6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bdd81081-065a-4e7e-af7d-d5937b216de6" (UID: "bdd81081-065a-4e7e-af7d-d5937b216de6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:24:26.424643 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:26.424610 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd81081-065a-4e7e-af7d-d5937b216de6-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:24:27.050656 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:27.050629 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-89hft" Jul 17 18:24:27.050828 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:27.050668 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-89hft" event={"ID":"bdd81081-065a-4e7e-af7d-d5937b216de6","Type":"ContainerDied","Data":"4ecfbec90e2857a29225af382d399ab02920a74dc7599712293a31d71d490a2d"} Jul 17 18:24:27.050828 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:27.050727 2574 scope.go:117] "RemoveContainer" containerID="0a9c97f4c07ff0af9040ab812144408910eee7310950f8e9d6ad08483f470b34" Jul 17 18:24:27.070648 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:27.070621 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-89hft"] Jul 17 18:24:27.074919 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:27.074887 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-89hft"] Jul 17 18:24:28.987658 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:28.987624 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdd81081-065a-4e7e-af7d-d5937b216de6" path="/var/lib/kubelet/pods/bdd81081-065a-4e7e-af7d-d5937b216de6/volumes" Jul 17 18:24:35.979440 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:35.979410 2574 scope.go:117] "RemoveContainer" containerID="c88356b4cb5d18fec822a4ef13dd235b732f96c6d4be0de0d016a478369625ab" Jul 17 18:24:37.113793 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:37.113754 2574 generic.go:356] "Generic (PLEG): container finished" podID="ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e" containerID="129be86c9e3923d3ea3cac58774e7f1b2d7a45dbad3ccacbc32bf24a588baf28" exitCode=7 Jul 17 18:24:37.114302 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:37.113831 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4" event={"ID":"ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e","Type":"ContainerDied","Data":"129be86c9e3923d3ea3cac58774e7f1b2d7a45dbad3ccacbc32bf24a588baf28"} Jul 17 18:24:37.114302 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:37.113923 2574 scope.go:117] "RemoveContainer" containerID="c88356b4cb5d18fec822a4ef13dd235b732f96c6d4be0de0d016a478369625ab" Jul 17 18:24:37.114302 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:37.114180 2574 scope.go:117] "RemoveContainer" containerID="129be86c9e3923d3ea3cac58774e7f1b2d7a45dbad3ccacbc32bf24a588baf28" Jul 17 18:24:37.114518 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:24:37.114427 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4_odh-ai-gateway-infra(ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4" podUID="ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e" Jul 17 18:24:38.141273 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:38.141238 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4"] Jul 17 18:24:38.280986 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:38.280956 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4" Jul 17 18:24:38.334737 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:38.334693 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e-maas-api-service-ca\") pod \"ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e\" (UID: \"ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e\") " Jul 17 18:24:38.335103 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:38.335075 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e" (UID: "ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:24:38.435627 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:38.435536 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:24:39.126650 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:39.126626 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4" Jul 17 18:24:39.126826 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:39.126618 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4" event={"ID":"ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e","Type":"ContainerDied","Data":"dc6501083b8a86791ad368a5fd249ba1e56493174937a571cde03665fb9ee884"} Jul 17 18:24:39.126826 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:39.126765 2574 scope.go:117] "RemoveContainer" containerID="129be86c9e3923d3ea3cac58774e7f1b2d7a45dbad3ccacbc32bf24a588baf28" Jul 17 18:24:39.143076 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:39.143047 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4"] Jul 17 18:24:39.146745 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:39.146723 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-w7ck4"] Jul 17 18:24:40.985674 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:40.985641 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e" path="/var/lib/kubelet/pods/ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e/volumes" Jul 17 18:24:42.989076 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:42.989042 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c"] Jul 17 18:24:42.989496 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:42.989444 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e" containerName="revoke-keys" Jul 17 18:24:42.989496 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:42.989456 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e" containerName="revoke-keys" Jul 17 18:24:42.989496 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:42.989474 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdd81081-065a-4e7e-af7d-d5937b216de6" containerName="revoke-keys" Jul 17 18:24:42.989496 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:42.989479 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdd81081-065a-4e7e-af7d-d5937b216de6" containerName="revoke-keys" Jul 17 18:24:42.989496 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:42.989486 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e" containerName="revoke-keys" Jul 17 18:24:42.989496 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:42.989492 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e" containerName="revoke-keys" Jul 17 18:24:42.989496 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:42.989497 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdd81081-065a-4e7e-af7d-d5937b216de6" containerName="revoke-keys" Jul 17 18:24:42.989715 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:42.989502 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdd81081-065a-4e7e-af7d-d5937b216de6" containerName="revoke-keys" Jul 17 18:24:42.989715 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:42.989512 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdd81081-065a-4e7e-af7d-d5937b216de6" containerName="revoke-keys" Jul 17 18:24:42.989715 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:42.989517 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdd81081-065a-4e7e-af7d-d5937b216de6" containerName="revoke-keys" Jul 17 18:24:42.989715 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:42.989573 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdd81081-065a-4e7e-af7d-d5937b216de6" containerName="revoke-keys" Jul 17 18:24:42.989715 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:42.989579 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e" containerName="revoke-keys" Jul 17 18:24:42.989715 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:42.989588 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdd81081-065a-4e7e-af7d-d5937b216de6" containerName="revoke-keys" Jul 17 18:24:42.989715 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:42.989595 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7e8de1d-74a0-4b63-b91f-712e2343fe6f" containerName="revoke-keys" Jul 17 18:24:42.989715 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:42.989601 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e" containerName="revoke-keys" Jul 17 18:24:42.994115 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:42.994096 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c" Jul 17 18:24:42.997694 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:42.997664 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dpsm\"" Jul 17 18:24:42.997845 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:42.997665 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:24:42.999079 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:42.999044 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c"] Jul 17 18:24:43.078249 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:43.078200 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5daaaee6-c218-4a0b-a7ee-27ea94f46ec3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c\" (UID: \"5daaaee6-c218-4a0b-a7ee-27ea94f46ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c" Jul 17 18:24:43.179470 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:43.179435 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5daaaee6-c218-4a0b-a7ee-27ea94f46ec3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c\" (UID: \"5daaaee6-c218-4a0b-a7ee-27ea94f46ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c" Jul 17 18:24:43.180090 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:43.180069 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5daaaee6-c218-4a0b-a7ee-27ea94f46ec3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c\" (UID: \"5daaaee6-c218-4a0b-a7ee-27ea94f46ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c" Jul 17 18:24:43.306814 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:43.306717 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c" Jul 17 18:24:43.466754 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:43.466727 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c"] Jul 17 18:24:43.469420 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:24:43.469389 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5daaaee6_c218_4a0b_a7ee_27ea94f46ec3.slice/crio-c90ce5079be72fcaaa68a79a2846a435fc21be9790657eb6d714002ee1f155e3 WatchSource:0}: Error finding container c90ce5079be72fcaaa68a79a2846a435fc21be9790657eb6d714002ee1f155e3: Status 404 returned error can't find the container with id c90ce5079be72fcaaa68a79a2846a435fc21be9790657eb6d714002ee1f155e3 Jul 17 18:24:44.156219 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:44.156188 2574 generic.go:356] "Generic (PLEG): container finished" podID="5daaaee6-c218-4a0b-a7ee-27ea94f46ec3" containerID="244820a2747c4d0f08d5a0ac04a23da03c94cac58d92c51b8c394a0292b4c6c3" exitCode=6 Jul 17 18:24:44.156632 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:44.156263 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c" event={"ID":"5daaaee6-c218-4a0b-a7ee-27ea94f46ec3","Type":"ContainerDied","Data":"244820a2747c4d0f08d5a0ac04a23da03c94cac58d92c51b8c394a0292b4c6c3"} Jul 17 18:24:44.156632 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:44.156301 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c" event={"ID":"5daaaee6-c218-4a0b-a7ee-27ea94f46ec3","Type":"ContainerStarted","Data":"c90ce5079be72fcaaa68a79a2846a435fc21be9790657eb6d714002ee1f155e3"} Jul 17 18:24:44.156632 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:44.156437 2574 scope.go:117] "RemoveContainer" containerID="244820a2747c4d0f08d5a0ac04a23da03c94cac58d92c51b8c394a0292b4c6c3" Jul 17 18:24:45.163786 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:45.163694 2574 generic.go:356] "Generic (PLEG): container finished" podID="5daaaee6-c218-4a0b-a7ee-27ea94f46ec3" containerID="9a327db79c94bf742bc8e57ea9f6f5aa2da660d0a915c5a49a872bf9f24e83de" exitCode=6 Jul 17 18:24:45.164212 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:45.163784 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c" event={"ID":"5daaaee6-c218-4a0b-a7ee-27ea94f46ec3","Type":"ContainerDied","Data":"9a327db79c94bf742bc8e57ea9f6f5aa2da660d0a915c5a49a872bf9f24e83de"} Jul 17 18:24:45.164212 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:45.163844 2574 scope.go:117] "RemoveContainer" containerID="244820a2747c4d0f08d5a0ac04a23da03c94cac58d92c51b8c394a0292b4c6c3" Jul 17 18:24:45.164212 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:45.163992 2574 scope.go:117] "RemoveContainer" containerID="9a327db79c94bf742bc8e57ea9f6f5aa2da660d0a915c5a49a872bf9f24e83de" Jul 17 18:24:45.164212 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:24:45.164203 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c_odh-ai-gateway-infra(5daaaee6-c218-4a0b-a7ee-27ea94f46ec3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c" podUID="5daaaee6-c218-4a0b-a7ee-27ea94f46ec3" Jul 17 18:24:46.170895 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:46.170861 2574 scope.go:117] "RemoveContainer" containerID="9a327db79c94bf742bc8e57ea9f6f5aa2da660d0a915c5a49a872bf9f24e83de" Jul 17 18:24:46.171435 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:24:46.171125 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c_odh-ai-gateway-infra(5daaaee6-c218-4a0b-a7ee-27ea94f46ec3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c" podUID="5daaaee6-c218-4a0b-a7ee-27ea94f46ec3" Jul 17 18:24:59.979824 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:24:59.979781 2574 scope.go:117] "RemoveContainer" containerID="9a327db79c94bf742bc8e57ea9f6f5aa2da660d0a915c5a49a872bf9f24e83de" Jul 17 18:25:00.806034 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:00.806001 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz"] Jul 17 18:25:00.806560 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:00.806540 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e" containerName="revoke-keys" Jul 17 18:25:00.806560 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:00.806562 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e" containerName="revoke-keys" Jul 17 18:25:00.806691 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:00.806640 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdd81081-065a-4e7e-af7d-d5937b216de6" containerName="revoke-keys" Jul 17 18:25:00.806691 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:00.806647 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce0df30f-3cec-4c59-8ee6-e3ed11d1ba7e" containerName="revoke-keys" Jul 17 18:25:00.809694 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:00.809678 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz" Jul 17 18:25:00.816468 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:00.816442 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz"] Jul 17 18:25:00.952672 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:00.952565 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d136efb9-0f0e-454e-8f6c-3c5b700e87c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz\" (UID: \"d136efb9-0f0e-454e-8f6c-3c5b700e87c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz" Jul 17 18:25:01.054120 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:01.054081 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d136efb9-0f0e-454e-8f6c-3c5b700e87c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz\" (UID: \"d136efb9-0f0e-454e-8f6c-3c5b700e87c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz" Jul 17 18:25:01.054783 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:01.054761 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d136efb9-0f0e-454e-8f6c-3c5b700e87c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz\" (UID: \"d136efb9-0f0e-454e-8f6c-3c5b700e87c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz" Jul 17 18:25:01.120941 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:01.120906 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz" Jul 17 18:25:01.250041 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:01.250002 2574 generic.go:356] "Generic (PLEG): container finished" podID="5daaaee6-c218-4a0b-a7ee-27ea94f46ec3" containerID="71e0d8daf52c3a4ac82e8b2d66dfada98ec8a74f4e5703f6a54b030b9765f587" exitCode=6 Jul 17 18:25:01.250208 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:01.250068 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c" event={"ID":"5daaaee6-c218-4a0b-a7ee-27ea94f46ec3","Type":"ContainerDied","Data":"71e0d8daf52c3a4ac82e8b2d66dfada98ec8a74f4e5703f6a54b030b9765f587"} Jul 17 18:25:01.250208 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:01.250133 2574 scope.go:117] "RemoveContainer" containerID="9a327db79c94bf742bc8e57ea9f6f5aa2da660d0a915c5a49a872bf9f24e83de" Jul 17 18:25:01.250443 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:01.250420 2574 scope.go:117] "RemoveContainer" containerID="71e0d8daf52c3a4ac82e8b2d66dfada98ec8a74f4e5703f6a54b030b9765f587" Jul 17 18:25:01.250658 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:25:01.250638 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c_odh-ai-gateway-infra(5daaaee6-c218-4a0b-a7ee-27ea94f46ec3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c" podUID="5daaaee6-c218-4a0b-a7ee-27ea94f46ec3" Jul 17 18:25:01.259698 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:01.259669 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz"] Jul 17 18:25:01.261510 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:25:01.261486 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd136efb9_0f0e_454e_8f6c_3c5b700e87c4.slice/crio-1d3716fcd7f13efe70db1714f493267cf8647d17fc832e482fc131a22d1690fa WatchSource:0}: Error finding container 1d3716fcd7f13efe70db1714f493267cf8647d17fc832e482fc131a22d1690fa: Status 404 returned error can't find the container with id 1d3716fcd7f13efe70db1714f493267cf8647d17fc832e482fc131a22d1690fa Jul 17 18:25:02.257713 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:02.257679 2574 generic.go:356] "Generic (PLEG): container finished" podID="d136efb9-0f0e-454e-8f6c-3c5b700e87c4" containerID="2f1eec288537ec599e6772276144b897b10b0f801b22e04ec60edc04123f5f22" exitCode=6 Jul 17 18:25:02.257713 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:02.257722 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz" event={"ID":"d136efb9-0f0e-454e-8f6c-3c5b700e87c4","Type":"ContainerDied","Data":"2f1eec288537ec599e6772276144b897b10b0f801b22e04ec60edc04123f5f22"} Jul 17 18:25:02.258165 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:02.257769 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz" event={"ID":"d136efb9-0f0e-454e-8f6c-3c5b700e87c4","Type":"ContainerStarted","Data":"1d3716fcd7f13efe70db1714f493267cf8647d17fc832e482fc131a22d1690fa"} Jul 17 18:25:02.258165 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:02.257982 2574 scope.go:117] "RemoveContainer" containerID="2f1eec288537ec599e6772276144b897b10b0f801b22e04ec60edc04123f5f22" Jul 17 18:25:02.276091 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:02.276056 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c"] Jul 17 18:25:02.421907 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:02.421880 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c" Jul 17 18:25:02.568653 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:02.568554 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5daaaee6-c218-4a0b-a7ee-27ea94f46ec3-maas-api-service-ca\") pod \"5daaaee6-c218-4a0b-a7ee-27ea94f46ec3\" (UID: \"5daaaee6-c218-4a0b-a7ee-27ea94f46ec3\") " Jul 17 18:25:02.568930 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:02.568908 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5daaaee6-c218-4a0b-a7ee-27ea94f46ec3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5daaaee6-c218-4a0b-a7ee-27ea94f46ec3" (UID: "5daaaee6-c218-4a0b-a7ee-27ea94f46ec3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:25:02.669806 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:02.669756 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5daaaee6-c218-4a0b-a7ee-27ea94f46ec3-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:25:03.265155 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:03.265066 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c" Jul 17 18:25:03.265155 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:03.265114 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c" event={"ID":"5daaaee6-c218-4a0b-a7ee-27ea94f46ec3","Type":"ContainerDied","Data":"c90ce5079be72fcaaa68a79a2846a435fc21be9790657eb6d714002ee1f155e3"} Jul 17 18:25:03.265705 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:03.265173 2574 scope.go:117] "RemoveContainer" containerID="71e0d8daf52c3a4ac82e8b2d66dfada98ec8a74f4e5703f6a54b030b9765f587" Jul 17 18:25:03.267652 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:03.267553 2574 generic.go:356] "Generic (PLEG): container finished" podID="d136efb9-0f0e-454e-8f6c-3c5b700e87c4" containerID="d11849a4110705c51847624620e56ec39a989354a743cc07026a6971fe237982" exitCode=6 Jul 17 18:25:03.267652 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:03.267629 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz" event={"ID":"d136efb9-0f0e-454e-8f6c-3c5b700e87c4","Type":"ContainerDied","Data":"d11849a4110705c51847624620e56ec39a989354a743cc07026a6971fe237982"} Jul 17 18:25:03.267851 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:03.267840 2574 scope.go:117] "RemoveContainer" containerID="d11849a4110705c51847624620e56ec39a989354a743cc07026a6971fe237982" Jul 17 18:25:03.268066 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:25:03.268046 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz_odh-ai-gateway-infra(d136efb9-0f0e-454e-8f6c-3c5b700e87c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz" podUID="d136efb9-0f0e-454e-8f6c-3c5b700e87c4" Jul 17 18:25:03.274447 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:03.274432 2574 scope.go:117] "RemoveContainer" containerID="2f1eec288537ec599e6772276144b897b10b0f801b22e04ec60edc04123f5f22" Jul 17 18:25:03.282007 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:03.281980 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c"] Jul 17 18:25:03.285563 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:03.285539 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-rvn4c"] Jul 17 18:25:04.276127 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:04.276100 2574 scope.go:117] "RemoveContainer" containerID="d11849a4110705c51847624620e56ec39a989354a743cc07026a6971fe237982" Jul 17 18:25:04.276565 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:25:04.276326 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz_odh-ai-gateway-infra(d136efb9-0f0e-454e-8f6c-3c5b700e87c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz" podUID="d136efb9-0f0e-454e-8f6c-3c5b700e87c4" Jul 17 18:25:04.985942 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:04.985896 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5daaaee6-c218-4a0b-a7ee-27ea94f46ec3" path="/var/lib/kubelet/pods/5daaaee6-c218-4a0b-a7ee-27ea94f46ec3/volumes" Jul 17 18:25:09.226964 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:09.226922 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k"] Jul 17 18:25:09.228421 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:09.228387 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5daaaee6-c218-4a0b-a7ee-27ea94f46ec3" containerName="revoke-keys" Jul 17 18:25:09.228421 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:09.228419 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5daaaee6-c218-4a0b-a7ee-27ea94f46ec3" containerName="revoke-keys" Jul 17 18:25:09.228654 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:09.228443 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5daaaee6-c218-4a0b-a7ee-27ea94f46ec3" containerName="revoke-keys" Jul 17 18:25:09.228654 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:09.228454 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5daaaee6-c218-4a0b-a7ee-27ea94f46ec3" containerName="revoke-keys" Jul 17 18:25:09.228654 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:09.228493 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5daaaee6-c218-4a0b-a7ee-27ea94f46ec3" containerName="revoke-keys" Jul 17 18:25:09.228654 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:09.228503 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5daaaee6-c218-4a0b-a7ee-27ea94f46ec3" containerName="revoke-keys" Jul 17 18:25:09.228847 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:09.228703 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5daaaee6-c218-4a0b-a7ee-27ea94f46ec3" containerName="revoke-keys" Jul 17 18:25:09.228847 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:09.228720 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5daaaee6-c218-4a0b-a7ee-27ea94f46ec3" containerName="revoke-keys" Jul 17 18:25:09.234640 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:09.234612 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k" Jul 17 18:25:09.235385 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:09.235334 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k"] Jul 17 18:25:09.333102 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:09.333064 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43036cc-1312-4210-87b3-3b7e2337c530-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad85eda1-f698k\" (UID: \"c43036cc-1312-4210-87b3-3b7e2337c530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k" Jul 17 18:25:09.433691 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:09.433637 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43036cc-1312-4210-87b3-3b7e2337c530-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad85eda1-f698k\" (UID: \"c43036cc-1312-4210-87b3-3b7e2337c530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k" Jul 17 18:25:09.434282 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:09.434262 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43036cc-1312-4210-87b3-3b7e2337c530-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad85eda1-f698k\" (UID: \"c43036cc-1312-4210-87b3-3b7e2337c530\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k" Jul 17 18:25:09.545541 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:09.545443 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k" Jul 17 18:25:09.679293 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:09.679263 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k"] Jul 17 18:25:09.681248 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:25:09.681216 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43036cc_1312_4210_87b3_3b7e2337c530.slice/crio-063141c838f5977930d86e6eefca05fea59cb01f93c3ea2d61a3c71023b3cd9e WatchSource:0}: Error finding container 063141c838f5977930d86e6eefca05fea59cb01f93c3ea2d61a3c71023b3cd9e: Status 404 returned error can't find the container with id 063141c838f5977930d86e6eefca05fea59cb01f93c3ea2d61a3c71023b3cd9e Jul 17 18:25:10.306564 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:10.306536 2574 generic.go:356] "Generic (PLEG): container finished" podID="c43036cc-1312-4210-87b3-3b7e2337c530" containerID="42723960e55e6f28f210ef6b87fdd6bd079a83e2c049708719663ed3fa09e9f4" exitCode=7 Jul 17 18:25:10.306920 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:10.306572 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k" event={"ID":"c43036cc-1312-4210-87b3-3b7e2337c530","Type":"ContainerDied","Data":"42723960e55e6f28f210ef6b87fdd6bd079a83e2c049708719663ed3fa09e9f4"} Jul 17 18:25:10.306920 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:10.306609 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k" event={"ID":"c43036cc-1312-4210-87b3-3b7e2337c530","Type":"ContainerStarted","Data":"063141c838f5977930d86e6eefca05fea59cb01f93c3ea2d61a3c71023b3cd9e"} Jul 17 18:25:10.306920 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:10.306811 2574 scope.go:117] "RemoveContainer" containerID="42723960e55e6f28f210ef6b87fdd6bd079a83e2c049708719663ed3fa09e9f4" Jul 17 18:25:11.313407 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:11.313278 2574 generic.go:356] "Generic (PLEG): container finished" podID="c43036cc-1312-4210-87b3-3b7e2337c530" containerID="ef520afaf03afd47e9b60aa7b6bb7daae303c10e8ffe9f4bed1885898e65ad72" exitCode=7 Jul 17 18:25:11.313407 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:11.313342 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k" event={"ID":"c43036cc-1312-4210-87b3-3b7e2337c530","Type":"ContainerDied","Data":"ef520afaf03afd47e9b60aa7b6bb7daae303c10e8ffe9f4bed1885898e65ad72"} Jul 17 18:25:11.313407 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:11.313409 2574 scope.go:117] "RemoveContainer" containerID="42723960e55e6f28f210ef6b87fdd6bd079a83e2c049708719663ed3fa09e9f4" Jul 17 18:25:11.313978 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:11.313650 2574 scope.go:117] "RemoveContainer" containerID="ef520afaf03afd47e9b60aa7b6bb7daae303c10e8ffe9f4bed1885898e65ad72" Jul 17 18:25:11.313978 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:25:11.313901 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ad85eda1-f698k_odh-ai-gateway-infra(c43036cc-1312-4210-87b3-3b7e2337c530)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k" podUID="c43036cc-1312-4210-87b3-3b7e2337c530" Jul 17 18:25:12.319889 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:12.319857 2574 scope.go:117] "RemoveContainer" containerID="ef520afaf03afd47e9b60aa7b6bb7daae303c10e8ffe9f4bed1885898e65ad72" Jul 17 18:25:12.320291 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:25:12.320058 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ad85eda1-f698k_odh-ai-gateway-infra(c43036cc-1312-4210-87b3-3b7e2337c530)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k" podUID="c43036cc-1312-4210-87b3-3b7e2337c530" Jul 17 18:25:17.980279 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:17.980244 2574 scope.go:117] "RemoveContainer" containerID="d11849a4110705c51847624620e56ec39a989354a743cc07026a6971fe237982" Jul 17 18:25:18.352909 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:18.352875 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz" event={"ID":"d136efb9-0f0e-454e-8f6c-3c5b700e87c4","Type":"ContainerStarted","Data":"39f30f4383e2f1e92190f4d6a252a238eb220ad3947499cbf6bbf33c28c13844"} Jul 17 18:25:18.371534 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:18.371478 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz" podStartSLOduration=18.371461197 podStartE2EDuration="18.371461197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:25:18.36925442 +0000 UTC m=+2129.957207120" watchObservedRunningTime="2026-07-17 18:25:18.371461197 +0000 UTC m=+2129.959413912" Jul 17 18:25:19.359924 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:19.359887 2574 generic.go:356] "Generic (PLEG): container finished" podID="d136efb9-0f0e-454e-8f6c-3c5b700e87c4" containerID="39f30f4383e2f1e92190f4d6a252a238eb220ad3947499cbf6bbf33c28c13844" exitCode=6 Jul 17 18:25:19.360380 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:19.359965 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz" event={"ID":"d136efb9-0f0e-454e-8f6c-3c5b700e87c4","Type":"ContainerDied","Data":"39f30f4383e2f1e92190f4d6a252a238eb220ad3947499cbf6bbf33c28c13844"} Jul 17 18:25:19.360380 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:19.360041 2574 scope.go:117] "RemoveContainer" containerID="d11849a4110705c51847624620e56ec39a989354a743cc07026a6971fe237982" Jul 17 18:25:19.360502 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:19.360416 2574 scope.go:117] "RemoveContainer" containerID="39f30f4383e2f1e92190f4d6a252a238eb220ad3947499cbf6bbf33c28c13844" Jul 17 18:25:19.360688 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:25:19.360665 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz_odh-ai-gateway-infra(d136efb9-0f0e-454e-8f6c-3c5b700e87c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz" podUID="d136efb9-0f0e-454e-8f6c-3c5b700e87c4" Jul 17 18:25:19.385253 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:19.385221 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz"] Jul 17 18:25:20.512484 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:20.512455 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz" Jul 17 18:25:20.644739 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:20.644636 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d136efb9-0f0e-454e-8f6c-3c5b700e87c4-maas-api-service-ca\") pod \"d136efb9-0f0e-454e-8f6c-3c5b700e87c4\" (UID: \"d136efb9-0f0e-454e-8f6c-3c5b700e87c4\") " Jul 17 18:25:20.645037 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:20.645014 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d136efb9-0f0e-454e-8f6c-3c5b700e87c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d136efb9-0f0e-454e-8f6c-3c5b700e87c4" (UID: "d136efb9-0f0e-454e-8f6c-3c5b700e87c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:25:20.745894 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:20.745859 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d136efb9-0f0e-454e-8f6c-3c5b700e87c4-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:25:21.374408 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:21.374299 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz" Jul 17 18:25:21.374408 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:21.374306 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz" event={"ID":"d136efb9-0f0e-454e-8f6c-3c5b700e87c4","Type":"ContainerDied","Data":"1d3716fcd7f13efe70db1714f493267cf8647d17fc832e482fc131a22d1690fa"} Jul 17 18:25:21.374644 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:21.374445 2574 scope.go:117] "RemoveContainer" containerID="39f30f4383e2f1e92190f4d6a252a238eb220ad3947499cbf6bbf33c28c13844" Jul 17 18:25:21.390859 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:21.390815 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz"] Jul 17 18:25:21.400587 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:21.400548 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-t6lzz"] Jul 17 18:25:22.980455 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:22.980419 2574 scope.go:117] "RemoveContainer" containerID="ef520afaf03afd47e9b60aa7b6bb7daae303c10e8ffe9f4bed1885898e65ad72" Jul 17 18:25:22.987309 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:22.987286 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d136efb9-0f0e-454e-8f6c-3c5b700e87c4" path="/var/lib/kubelet/pods/d136efb9-0f0e-454e-8f6c-3c5b700e87c4/volumes" Jul 17 18:25:23.389192 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:23.389157 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k" event={"ID":"c43036cc-1312-4210-87b3-3b7e2337c530","Type":"ContainerStarted","Data":"c74156d3aa65b69a6aa5e550bb5a89763d37f29faa6cc6f8af0101f0f220a5c0"} Jul 17 18:25:23.405785 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:23.405720 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k" podStartSLOduration=14.40569928 podStartE2EDuration="14.40569928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:25:23.405177192 +0000 UTC m=+2134.993129905" watchObservedRunningTime="2026-07-17 18:25:23.40569928 +0000 UTC m=+2134.993651971" Jul 17 18:25:24.396828 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:24.396798 2574 generic.go:356] "Generic (PLEG): container finished" podID="c43036cc-1312-4210-87b3-3b7e2337c530" containerID="c74156d3aa65b69a6aa5e550bb5a89763d37f29faa6cc6f8af0101f0f220a5c0" exitCode=7 Jul 17 18:25:24.397298 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:24.396877 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k" event={"ID":"c43036cc-1312-4210-87b3-3b7e2337c530","Type":"ContainerDied","Data":"c74156d3aa65b69a6aa5e550bb5a89763d37f29faa6cc6f8af0101f0f220a5c0"} Jul 17 18:25:24.397298 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:24.396955 2574 scope.go:117] "RemoveContainer" containerID="ef520afaf03afd47e9b60aa7b6bb7daae303c10e8ffe9f4bed1885898e65ad72" Jul 17 18:25:24.397468 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:24.397306 2574 scope.go:117] "RemoveContainer" containerID="c74156d3aa65b69a6aa5e550bb5a89763d37f29faa6cc6f8af0101f0f220a5c0" Jul 17 18:25:24.397563 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:25:24.397545 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ad85eda1-f698k_odh-ai-gateway-infra(c43036cc-1312-4210-87b3-3b7e2337c530)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k" podUID="c43036cc-1312-4210-87b3-3b7e2337c530" Jul 17 18:25:24.420337 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:24.420307 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k"] Jul 17 18:25:25.553238 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:25.553209 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k" Jul 17 18:25:25.587402 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:25.587345 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43036cc-1312-4210-87b3-3b7e2337c530-maas-api-service-ca\") pod \"c43036cc-1312-4210-87b3-3b7e2337c530\" (UID: \"c43036cc-1312-4210-87b3-3b7e2337c530\") " Jul 17 18:25:25.587717 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:25.587689 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c43036cc-1312-4210-87b3-3b7e2337c530-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c43036cc-1312-4210-87b3-3b7e2337c530" (UID: "c43036cc-1312-4210-87b3-3b7e2337c530"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:25:25.688406 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:25.688294 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43036cc-1312-4210-87b3-3b7e2337c530-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:25:26.410060 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:26.410030 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k" Jul 17 18:25:26.410060 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:26.410053 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k" event={"ID":"c43036cc-1312-4210-87b3-3b7e2337c530","Type":"ContainerDied","Data":"063141c838f5977930d86e6eefca05fea59cb01f93c3ea2d61a3c71023b3cd9e"} Jul 17 18:25:26.410321 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:26.410108 2574 scope.go:117] "RemoveContainer" containerID="c74156d3aa65b69a6aa5e550bb5a89763d37f29faa6cc6f8af0101f0f220a5c0" Jul 17 18:25:26.433368 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:26.433327 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k"] Jul 17 18:25:26.442562 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:26.442532 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad85eda1-f698k"] Jul 17 18:25:26.985789 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:26.985753 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c43036cc-1312-4210-87b3-3b7e2337c530" path="/var/lib/kubelet/pods/c43036cc-1312-4210-87b3-3b7e2337c530/volumes" Jul 17 18:25:29.288130 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:29.288084 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:25:29.351807 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:29.351771 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:25:34.720449 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:34.720414 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:25:35.247955 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:35.247913 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:25:40.664052 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:40.663958 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:25:40.762875 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:40.762833 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:25:41.147830 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:41.147791 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8f9c7c886-rc5n9_17141199-6d8a-4bcd-be1b-c00b1b7c6b29/authorino/0.log" Jul 17 18:25:43.026989 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.026942 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q"] Jul 17 18:25:43.027427 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.027394 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d136efb9-0f0e-454e-8f6c-3c5b700e87c4" containerName="revoke-keys" Jul 17 18:25:43.027427 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.027409 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d136efb9-0f0e-454e-8f6c-3c5b700e87c4" containerName="revoke-keys" Jul 17 18:25:43.027427 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.027418 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d136efb9-0f0e-454e-8f6c-3c5b700e87c4" containerName="revoke-keys" Jul 17 18:25:43.027427 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.027424 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d136efb9-0f0e-454e-8f6c-3c5b700e87c4" containerName="revoke-keys" Jul 17 18:25:43.027598 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.027438 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c43036cc-1312-4210-87b3-3b7e2337c530" containerName="revoke-keys" Jul 17 18:25:43.027598 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.027444 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c43036cc-1312-4210-87b3-3b7e2337c530" containerName="revoke-keys" Jul 17 18:25:43.027598 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.027457 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c43036cc-1312-4210-87b3-3b7e2337c530" containerName="revoke-keys" Jul 17 18:25:43.027598 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.027464 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c43036cc-1312-4210-87b3-3b7e2337c530" containerName="revoke-keys" Jul 17 18:25:43.027598 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.027531 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d136efb9-0f0e-454e-8f6c-3c5b700e87c4" containerName="revoke-keys" Jul 17 18:25:43.027598 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.027539 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d136efb9-0f0e-454e-8f6c-3c5b700e87c4" containerName="revoke-keys" Jul 17 18:25:43.027598 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.027545 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d136efb9-0f0e-454e-8f6c-3c5b700e87c4" containerName="revoke-keys" Jul 17 18:25:43.027598 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.027551 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5daaaee6-c218-4a0b-a7ee-27ea94f46ec3" containerName="revoke-keys" Jul 17 18:25:43.027598 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.027556 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c43036cc-1312-4210-87b3-3b7e2337c530" containerName="revoke-keys" Jul 17 18:25:43.027598 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.027562 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c43036cc-1312-4210-87b3-3b7e2337c530" containerName="revoke-keys" Jul 17 18:25:43.027598 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.027569 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c43036cc-1312-4210-87b3-3b7e2337c530" containerName="revoke-keys" Jul 17 18:25:43.032172 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.032151 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q" Jul 17 18:25:43.036455 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.036240 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dpsm\"" Jul 17 18:25:43.036455 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.036256 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:25:43.038066 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.038039 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q"] Jul 17 18:25:43.157677 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.157638 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9d56dd-db1f-4fd7-98c9-efae0226fd4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q\" (UID: \"4f9d56dd-db1f-4fd7-98c9-efae0226fd4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q" Jul 17 18:25:43.258410 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.258334 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9d56dd-db1f-4fd7-98c9-efae0226fd4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q\" (UID: \"4f9d56dd-db1f-4fd7-98c9-efae0226fd4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q" Jul 17 18:25:43.258976 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.258956 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9d56dd-db1f-4fd7-98c9-efae0226fd4a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q\" (UID: \"4f9d56dd-db1f-4fd7-98c9-efae0226fd4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q" Jul 17 18:25:43.344072 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.343964 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q" Jul 17 18:25:43.484317 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.484288 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q"] Jul 17 18:25:43.486309 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:25:43.486281 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9d56dd_db1f_4fd7_98c9_efae0226fd4a.slice/crio-e3329cf05585a3922db11c1aa21a47524874e42e18a61c5e4585e1c30d4d5245 WatchSource:0}: Error finding container e3329cf05585a3922db11c1aa21a47524874e42e18a61c5e4585e1c30d4d5245: Status 404 returned error can't find the container with id e3329cf05585a3922db11c1aa21a47524874e42e18a61c5e4585e1c30d4d5245 Jul 17 18:25:43.502995 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:43.502961 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q" event={"ID":"4f9d56dd-db1f-4fd7-98c9-efae0226fd4a","Type":"ContainerStarted","Data":"e3329cf05585a3922db11c1aa21a47524874e42e18a61c5e4585e1c30d4d5245"} Jul 17 18:25:44.509981 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:44.509946 2574 generic.go:356] "Generic (PLEG): container finished" podID="4f9d56dd-db1f-4fd7-98c9-efae0226fd4a" containerID="03e5b5caa8455c8f699f830c8ea19d283aeb5b9d7523a90b3d508cfcac3cf51f" exitCode=6 Jul 17 18:25:44.510437 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:44.510032 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q" event={"ID":"4f9d56dd-db1f-4fd7-98c9-efae0226fd4a","Type":"ContainerDied","Data":"03e5b5caa8455c8f699f830c8ea19d283aeb5b9d7523a90b3d508cfcac3cf51f"} Jul 17 18:25:44.510437 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:44.510204 2574 scope.go:117] "RemoveContainer" containerID="03e5b5caa8455c8f699f830c8ea19d283aeb5b9d7523a90b3d508cfcac3cf51f" Jul 17 18:25:45.517892 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:45.517853 2574 generic.go:356] "Generic (PLEG): container finished" podID="4f9d56dd-db1f-4fd7-98c9-efae0226fd4a" containerID="e2b30a2148573e80d63a7484094938426b2b56c82ae8b966a91282a0369bb675" exitCode=6 Jul 17 18:25:45.518370 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:45.517938 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q" event={"ID":"4f9d56dd-db1f-4fd7-98c9-efae0226fd4a","Type":"ContainerDied","Data":"e2b30a2148573e80d63a7484094938426b2b56c82ae8b966a91282a0369bb675"} Jul 17 18:25:45.518370 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:45.518025 2574 scope.go:117] "RemoveContainer" containerID="03e5b5caa8455c8f699f830c8ea19d283aeb5b9d7523a90b3d508cfcac3cf51f" Jul 17 18:25:45.518370 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:45.518217 2574 scope.go:117] "RemoveContainer" containerID="e2b30a2148573e80d63a7484094938426b2b56c82ae8b966a91282a0369bb675" Jul 17 18:25:45.518512 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:25:45.518492 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q_odh-ai-gateway-infra(4f9d56dd-db1f-4fd7-98c9-efae0226fd4a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q" podUID="4f9d56dd-db1f-4fd7-98c9-efae0226fd4a" Jul 17 18:25:46.525107 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:46.525080 2574 scope.go:117] "RemoveContainer" containerID="e2b30a2148573e80d63a7484094938426b2b56c82ae8b966a91282a0369bb675" Jul 17 18:25:46.525615 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:25:46.525407 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q_odh-ai-gateway-infra(4f9d56dd-db1f-4fd7-98c9-efae0226fd4a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q" podUID="4f9d56dd-db1f-4fd7-98c9-efae0226fd4a" Jul 17 18:25:47.007309 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:47.007276 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cc99fdf4f-bmtvp_7f8367df-920a-417e-8033-8bad870315c6/copy-manifests/0.log" Jul 17 18:25:47.014748 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:47.014716 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cc99fdf4f-bmtvp_7f8367df-920a-417e-8033-8bad870315c6/manager/0.log" Jul 17 18:25:47.687554 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:47.687523 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738520-929bz_2a161bd7-b2ed-4618-bfcc-528b681b9353/cleanup/0.log" Jul 17 18:25:47.789569 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:47.789532 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738535-d9vlp_d448bbf4-fa79-48e3-9312-b2e64d74478c/cleanup/0.log" Jul 17 18:25:47.895966 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:47.895940 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q_4f9d56dd-db1f-4fd7-98c9-efae0226fd4a/revoke-keys/1.log" Jul 17 18:25:49.247024 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:49.246987 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8f9c7c886-rc5n9_17141199-6d8a-4bcd-be1b-c00b1b7c6b29/authorino/0.log" Jul 17 18:25:49.823868 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:49.823815 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-s242v_00dc5a78-7403-4a3f-850e-1ec3f63a6e6d/manager/0.log" Jul 17 18:25:50.767529 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:50.767476 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6799b9db4b-nhjjk_eb3196bd-adc3-43b7-a538-9575d48afa1a/kube-auth-proxy/0.log" Jul 17 18:25:51.017850 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:51.017747 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-xbqzn_c483e1a0-1ed1-4f87-a9c9-aa84e01a72f2/payload-pre-processing/0.log" Jul 17 18:25:51.611947 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:51.611910 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-j284m_875c5d7b-aa74-459a-a3fe-8b9bac191aea/main/0.log" Jul 17 18:25:55.872127 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:55.872092 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:25:56.979695 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:56.979656 2574 scope.go:117] "RemoveContainer" containerID="e2b30a2148573e80d63a7484094938426b2b56c82ae8b966a91282a0369bb675" Jul 17 18:25:57.584684 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:57.584645 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q" event={"ID":"4f9d56dd-db1f-4fd7-98c9-efae0226fd4a","Type":"ContainerStarted","Data":"8b2d4ccbe5cf82002ef7fe0e35e8e00edb7021746a4e391d9bf01e1fac10f54a"} Jul 17 18:25:57.602961 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:57.602899 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q" podStartSLOduration=14.60287908 podStartE2EDuration="14.60287908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:25:57.602596981 +0000 UTC m=+2169.190549695" watchObservedRunningTime="2026-07-17 18:25:57.60287908 +0000 UTC m=+2169.190831771" Jul 17 18:25:57.898261 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:57.898171 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wdmc6/must-gather-c96xv"] Jul 17 18:25:57.898613 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:57.898599 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d136efb9-0f0e-454e-8f6c-3c5b700e87c4" containerName="revoke-keys" Jul 17 18:25:57.898667 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:57.898617 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d136efb9-0f0e-454e-8f6c-3c5b700e87c4" containerName="revoke-keys" Jul 17 18:25:57.898667 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:57.898634 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c43036cc-1312-4210-87b3-3b7e2337c530" containerName="revoke-keys" Jul 17 18:25:57.898667 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:57.898639 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c43036cc-1312-4210-87b3-3b7e2337c530" containerName="revoke-keys" Jul 17 18:25:57.900989 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:57.900962 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:25:57.901896 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:57.901876 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wdmc6/must-gather-c96xv" Jul 17 18:25:57.904633 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:57.904602 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wdmc6\"/\"default-dockercfg-87g6t\"" Jul 17 18:25:57.904770 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:57.904657 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wdmc6\"/\"kube-root-ca.crt\"" Jul 17 18:25:57.904770 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:57.904615 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wdmc6\"/\"openshift-service-ca.crt\"" Jul 17 18:25:57.909513 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:57.909490 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wdmc6/must-gather-c96xv"] Jul 17 18:25:57.998389 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:57.998323 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/985e4ba3-d957-4533-958e-89146fa77903-must-gather-output\") pod \"must-gather-c96xv\" (UID: \"985e4ba3-d957-4533-958e-89146fa77903\") " pod="openshift-must-gather-wdmc6/must-gather-c96xv" Jul 17 18:25:57.998836 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:57.998504 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67x99\" (UniqueName: \"kubernetes.io/projected/985e4ba3-d957-4533-958e-89146fa77903-kube-api-access-67x99\") pod \"must-gather-c96xv\" (UID: \"985e4ba3-d957-4533-958e-89146fa77903\") " pod="openshift-must-gather-wdmc6/must-gather-c96xv" Jul 17 18:25:58.000897 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:58.000860 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" limit=1024 Jul 17 18:25:58.100036 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:58.099992 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67x99\" (UniqueName: \"kubernetes.io/projected/985e4ba3-d957-4533-958e-89146fa77903-kube-api-access-67x99\") pod \"must-gather-c96xv\" (UID: \"985e4ba3-d957-4533-958e-89146fa77903\") " pod="openshift-must-gather-wdmc6/must-gather-c96xv" Jul 17 18:25:58.100226 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:58.100072 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/985e4ba3-d957-4533-958e-89146fa77903-must-gather-output\") pod \"must-gather-c96xv\" (UID: \"985e4ba3-d957-4533-958e-89146fa77903\") " pod="openshift-must-gather-wdmc6/must-gather-c96xv" Jul 17 18:25:58.100425 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:58.100409 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/985e4ba3-d957-4533-958e-89146fa77903-must-gather-output\") pod \"must-gather-c96xv\" (UID: \"985e4ba3-d957-4533-958e-89146fa77903\") " pod="openshift-must-gather-wdmc6/must-gather-c96xv" Jul 17 18:25:58.107967 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:58.107930 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67x99\" (UniqueName: \"kubernetes.io/projected/985e4ba3-d957-4533-958e-89146fa77903-kube-api-access-67x99\") pod \"must-gather-c96xv\" (UID: \"985e4ba3-d957-4533-958e-89146fa77903\") " pod="openshift-must-gather-wdmc6/must-gather-c96xv" Jul 17 18:25:58.212371 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:58.212260 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wdmc6/must-gather-c96xv" Jul 17 18:25:58.357749 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:58.357722 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wdmc6/must-gather-c96xv"] Jul 17 18:25:58.359598 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:25:58.359570 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod985e4ba3_d957_4533_958e_89146fa77903.slice/crio-9069ea1d845ae14797c19a9094cdb2ba45cf8215007f7efe805733584a3d51ee WatchSource:0}: Error finding container 9069ea1d845ae14797c19a9094cdb2ba45cf8215007f7efe805733584a3d51ee: Status 404 returned error can't find the container with id 9069ea1d845ae14797c19a9094cdb2ba45cf8215007f7efe805733584a3d51ee Jul 17 18:25:58.361743 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:58.361727 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:25:58.591809 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:58.591777 2574 generic.go:356] "Generic (PLEG): container finished" podID="4f9d56dd-db1f-4fd7-98c9-efae0226fd4a" containerID="8b2d4ccbe5cf82002ef7fe0e35e8e00edb7021746a4e391d9bf01e1fac10f54a" exitCode=6 Jul 17 18:25:58.591985 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:58.591847 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q" event={"ID":"4f9d56dd-db1f-4fd7-98c9-efae0226fd4a","Type":"ContainerDied","Data":"8b2d4ccbe5cf82002ef7fe0e35e8e00edb7021746a4e391d9bf01e1fac10f54a"} Jul 17 18:25:58.591985 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:58.591908 2574 scope.go:117] "RemoveContainer" containerID="e2b30a2148573e80d63a7484094938426b2b56c82ae8b966a91282a0369bb675" Jul 17 18:25:58.592262 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:58.592244 2574 scope.go:117] "RemoveContainer" containerID="8b2d4ccbe5cf82002ef7fe0e35e8e00edb7021746a4e391d9bf01e1fac10f54a" Jul 17 18:25:58.592549 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:25:58.592531 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q_odh-ai-gateway-infra(4f9d56dd-db1f-4fd7-98c9-efae0226fd4a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q" podUID="4f9d56dd-db1f-4fd7-98c9-efae0226fd4a" Jul 17 18:25:58.593157 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:58.593135 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wdmc6/must-gather-c96xv" event={"ID":"985e4ba3-d957-4533-958e-89146fa77903","Type":"ContainerStarted","Data":"9069ea1d845ae14797c19a9094cdb2ba45cf8215007f7efe805733584a3d51ee"} Jul 17 18:25:58.614348 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:58.614318 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q"] Jul 17 18:25:59.604256 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:59.604219 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wdmc6/must-gather-c96xv" event={"ID":"985e4ba3-d957-4533-958e-89146fa77903","Type":"ContainerStarted","Data":"dd597788fc8c3e4bf24eada7729a48cd7c7b2c532642e5f84557cb397c69e317"} Jul 17 18:25:59.604694 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:59.604272 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wdmc6/must-gather-c96xv" event={"ID":"985e4ba3-d957-4533-958e-89146fa77903","Type":"ContainerStarted","Data":"faaac012620fbfffb65288cd609366016e657a54094861f77137e3a90889945a"} Jul 17 18:25:59.632016 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:59.631958 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wdmc6/must-gather-c96xv" podStartSLOduration=1.788088138 podStartE2EDuration="2.631940975s" totalImagesPullingTime="843.852837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:25:59.631310423 +0000 UTC m=+2171.219263115" watchObservedRunningTime="2026-07-17 18:25:59.631940975 +0000 UTC m=+2171.219893664" Jul 17 18:25:59.778952 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:59.778920 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q" Jul 17 18:25:59.918666 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:59.918583 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9d56dd-db1f-4fd7-98c9-efae0226fd4a-maas-api-service-ca\") pod \"4f9d56dd-db1f-4fd7-98c9-efae0226fd4a\" (UID: \"4f9d56dd-db1f-4fd7-98c9-efae0226fd4a\") " Jul 17 18:25:59.918935 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:25:59.918909 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f9d56dd-db1f-4fd7-98c9-efae0226fd4a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4f9d56dd-db1f-4fd7-98c9-efae0226fd4a" (UID: "4f9d56dd-db1f-4fd7-98c9-efae0226fd4a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:26:00.020182 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.020142 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f9d56dd-db1f-4fd7-98c9-efae0226fd4a-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:26:00.611676 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.611636 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q" event={"ID":"4f9d56dd-db1f-4fd7-98c9-efae0226fd4a","Type":"ContainerDied","Data":"e3329cf05585a3922db11c1aa21a47524874e42e18a61c5e4585e1c30d4d5245"} Jul 17 18:26:00.612164 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.611709 2574 scope.go:117] "RemoveContainer" containerID="8b2d4ccbe5cf82002ef7fe0e35e8e00edb7021746a4e391d9bf01e1fac10f54a" Jul 17 18:26:00.612164 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.611655 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q" Jul 17 18:26:00.639107 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.639069 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q"] Jul 17 18:26:00.645034 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.644996 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-284e9f2b-wlv4q"] Jul 17 18:26:00.800070 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.800039 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4xjqt_fa00821b-8e34-4030-9988-e37c6a12230d/global-pull-secret-syncer/0.log" Jul 17 18:26:00.846553 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.846515 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j"] Jul 17 18:26:00.846935 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.846919 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f9d56dd-db1f-4fd7-98c9-efae0226fd4a" containerName="revoke-keys" Jul 17 18:26:00.846935 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.846933 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f9d56dd-db1f-4fd7-98c9-efae0226fd4a" containerName="revoke-keys" Jul 17 18:26:00.847097 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.846940 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f9d56dd-db1f-4fd7-98c9-efae0226fd4a" containerName="revoke-keys" Jul 17 18:26:00.847097 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.846946 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f9d56dd-db1f-4fd7-98c9-efae0226fd4a" containerName="revoke-keys" Jul 17 18:26:00.847097 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.846955 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f9d56dd-db1f-4fd7-98c9-efae0226fd4a" containerName="revoke-keys" Jul 17 18:26:00.847097 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.846960 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f9d56dd-db1f-4fd7-98c9-efae0226fd4a" containerName="revoke-keys" Jul 17 18:26:00.847097 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.847013 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f9d56dd-db1f-4fd7-98c9-efae0226fd4a" containerName="revoke-keys" Jul 17 18:26:00.847097 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.847020 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f9d56dd-db1f-4fd7-98c9-efae0226fd4a" containerName="revoke-keys" Jul 17 18:26:00.851938 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.851896 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j" Jul 17 18:26:00.854896 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.854781 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dpsm\"" Jul 17 18:26:00.855100 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.854989 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 18:26:00.861408 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.861377 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j"] Jul 17 18:26:00.929382 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.929239 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f213ae2-3c83-45be-a3c8-32314c9d5162-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j\" (UID: \"3f213ae2-3c83-45be-a3c8-32314c9d5162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j" Jul 17 18:26:00.947607 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.947575 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-db2n2_3a6a8a84-5e48-4b57-b683-a29619e9120f/konnectivity-agent/0.log" Jul 17 18:26:00.988028 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:00.987999 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f9d56dd-db1f-4fd7-98c9-efae0226fd4a" path="/var/lib/kubelet/pods/4f9d56dd-db1f-4fd7-98c9-efae0226fd4a/volumes" Jul 17 18:26:01.030791 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:01.030755 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f213ae2-3c83-45be-a3c8-32314c9d5162-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j\" (UID: \"3f213ae2-3c83-45be-a3c8-32314c9d5162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j" Jul 17 18:26:01.031347 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:01.031328 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f213ae2-3c83-45be-a3c8-32314c9d5162-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j\" (UID: \"3f213ae2-3c83-45be-a3c8-32314c9d5162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j" Jul 17 18:26:01.087190 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:01.087159 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-155.ec2.internal_7aea52351981b999e804455ac7eec0e6/haproxy/0.log" Jul 17 18:26:01.165961 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:01.165910 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j" Jul 17 18:26:01.343860 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:01.343551 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j"] Jul 17 18:26:01.620505 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:01.620459 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j" event={"ID":"3f213ae2-3c83-45be-a3c8-32314c9d5162","Type":"ContainerStarted","Data":"d7ad4eb556724b37409bced37ff8d6df8ce62cd48b1a0d0abde9c7d9c4cd17e2"} Jul 17 18:26:02.638367 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:02.635788 2574 generic.go:356] "Generic (PLEG): container finished" podID="3f213ae2-3c83-45be-a3c8-32314c9d5162" containerID="5e048c8df11039e38b7e0a6de7016fd9a4758109d06124d29244355aa129103d" exitCode=6 Jul 17 18:26:02.638367 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:02.635941 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j" event={"ID":"3f213ae2-3c83-45be-a3c8-32314c9d5162","Type":"ContainerDied","Data":"5e048c8df11039e38b7e0a6de7016fd9a4758109d06124d29244355aa129103d"} Jul 17 18:26:02.638367 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:02.636329 2574 scope.go:117] "RemoveContainer" containerID="5e048c8df11039e38b7e0a6de7016fd9a4758109d06124d29244355aa129103d" Jul 17 18:26:03.648305 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:03.648255 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j" event={"ID":"3f213ae2-3c83-45be-a3c8-32314c9d5162","Type":"ContainerStarted","Data":"3253fc13db80124907026ff60212b08c27e446099ff78f01ed8aa907fd7240c9"} Jul 17 18:26:04.660036 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:04.658751 2574 generic.go:356] "Generic (PLEG): container finished" podID="3f213ae2-3c83-45be-a3c8-32314c9d5162" containerID="3253fc13db80124907026ff60212b08c27e446099ff78f01ed8aa907fd7240c9" exitCode=6 Jul 17 18:26:04.660036 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:04.658801 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j" event={"ID":"3f213ae2-3c83-45be-a3c8-32314c9d5162","Type":"ContainerDied","Data":"3253fc13db80124907026ff60212b08c27e446099ff78f01ed8aa907fd7240c9"} Jul 17 18:26:04.660036 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:04.658860 2574 scope.go:117] "RemoveContainer" containerID="5e048c8df11039e38b7e0a6de7016fd9a4758109d06124d29244355aa129103d" Jul 17 18:26:04.660036 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:04.659212 2574 scope.go:117] "RemoveContainer" containerID="3253fc13db80124907026ff60212b08c27e446099ff78f01ed8aa907fd7240c9" Jul 17 18:26:04.660036 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:26:04.659480 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j_odh-ai-gateway-infra(3f213ae2-3c83-45be-a3c8-32314c9d5162)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j" podUID="3f213ae2-3c83-45be-a3c8-32314c9d5162" Jul 17 18:26:05.670886 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:05.670066 2574 scope.go:117] "RemoveContainer" containerID="3253fc13db80124907026ff60212b08c27e446099ff78f01ed8aa907fd7240c9" Jul 17 18:26:05.670886 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:26:05.670338 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j_odh-ai-gateway-infra(3f213ae2-3c83-45be-a3c8-32314c9d5162)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j" podUID="3f213ae2-3c83-45be-a3c8-32314c9d5162" Jul 17 18:26:06.266375 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:06.265743 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8f9c7c886-rc5n9_17141199-6d8a-4bcd-be1b-c00b1b7c6b29/authorino/0.log" Jul 17 18:26:06.528771 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:06.528623 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-s242v_00dc5a78-7403-4a3f-850e-1ec3f63a6e6d/manager/0.log" Jul 17 18:26:08.349918 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:08.349834 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c0ab0b93-d246-49b1-bddd-f1bcba272d12/alertmanager/0.log" Jul 17 18:26:08.384909 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:08.384881 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c0ab0b93-d246-49b1-bddd-f1bcba272d12/config-reloader/0.log" Jul 17 18:26:08.409560 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:08.409534 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c0ab0b93-d246-49b1-bddd-f1bcba272d12/kube-rbac-proxy-web/0.log" Jul 17 18:26:08.437648 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:08.437611 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c0ab0b93-d246-49b1-bddd-f1bcba272d12/kube-rbac-proxy/0.log" Jul 17 18:26:08.463103 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:08.463065 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c0ab0b93-d246-49b1-bddd-f1bcba272d12/kube-rbac-proxy-metric/0.log" Jul 17 18:26:08.490011 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:08.489984 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c0ab0b93-d246-49b1-bddd-f1bcba272d12/prom-label-proxy/0.log" Jul 17 18:26:08.530497 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:08.530473 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c0ab0b93-d246-49b1-bddd-f1bcba272d12/init-config-reloader/0.log" Jul 17 18:26:08.613836 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:08.613739 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-hbxng_90507912-ab72-4d0a-927e-610d3b2ac868/kube-state-metrics/0.log" Jul 17 18:26:08.641705 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:08.641674 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-hbxng_90507912-ab72-4d0a-927e-610d3b2ac868/kube-rbac-proxy-main/0.log" Jul 17 18:26:08.675120 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:08.675024 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-hbxng_90507912-ab72-4d0a-927e-610d3b2ac868/kube-rbac-proxy-self/0.log" Jul 17 18:26:08.716046 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:08.716004 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-76c895f64f-2ps4v_42b1b22b-8e4e-4b3d-a28a-9b2aacb0982a/metrics-server/0.log" Jul 17 18:26:08.752271 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:08.752245 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-wcdrz_41c7de63-1216-4c58-ba85-53c3dfe70641/monitoring-plugin/0.log" Jul 17 18:26:08.971241 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:08.971203 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zdsq2_59f60f6b-32d4-477b-a263-9c96b3eac478/node-exporter/0.log" Jul 17 18:26:08.998753 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:08.998709 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zdsq2_59f60f6b-32d4-477b-a263-9c96b3eac478/kube-rbac-proxy/0.log" Jul 17 18:26:09.023771 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:09.023735 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zdsq2_59f60f6b-32d4-477b-a263-9c96b3eac478/init-textfile/0.log" Jul 17 18:26:09.053874 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:09.053843 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-cj4hd_1e69c15b-cecf-46ca-99a7-4dd44d4b8c02/kube-rbac-proxy-main/0.log" Jul 17 18:26:09.075551 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:09.075519 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-cj4hd_1e69c15b-cecf-46ca-99a7-4dd44d4b8c02/kube-rbac-proxy-self/0.log" Jul 17 18:26:09.098184 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:09.098150 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-cj4hd_1e69c15b-cecf-46ca-99a7-4dd44d4b8c02/openshift-state-metrics/0.log" Jul 17 18:26:09.137331 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:09.137288 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_04402fe8-4d27-4800-b9c4-ebccfa4a2c4f/prometheus/0.log" Jul 17 18:26:09.165006 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:09.164977 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_04402fe8-4d27-4800-b9c4-ebccfa4a2c4f/config-reloader/0.log" Jul 17 18:26:09.191098 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:09.191008 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_04402fe8-4d27-4800-b9c4-ebccfa4a2c4f/thanos-sidecar/0.log" Jul 17 18:26:09.221060 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:09.221025 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_04402fe8-4d27-4800-b9c4-ebccfa4a2c4f/kube-rbac-proxy-web/0.log" Jul 17 18:26:09.245196 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:09.245165 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_04402fe8-4d27-4800-b9c4-ebccfa4a2c4f/kube-rbac-proxy/0.log" Jul 17 18:26:09.267664 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:09.267631 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_04402fe8-4d27-4800-b9c4-ebccfa4a2c4f/kube-rbac-proxy-thanos/0.log" Jul 17 18:26:09.292398 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:09.292338 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_04402fe8-4d27-4800-b9c4-ebccfa4a2c4f/init-config-reloader/0.log" Jul 17 18:26:09.324833 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:09.324801 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-nglj8_f34dcf01-3e4a-4d84-9e43-db829b532671/prometheus-operator/0.log" Jul 17 18:26:09.351148 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:09.351104 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-nglj8_f34dcf01-3e4a-4d84-9e43-db829b532671/kube-rbac-proxy/0.log" Jul 17 18:26:09.408993 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:09.408965 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c4484c75-w6cpp_45700789-5a8a-4b85-a437-abdb1591e11e/telemeter-client/0.log" Jul 17 18:26:09.431550 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:09.431523 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c4484c75-w6cpp_45700789-5a8a-4b85-a437-abdb1591e11e/reload/0.log" Jul 17 18:26:09.453754 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:09.453672 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c4484c75-w6cpp_45700789-5a8a-4b85-a437-abdb1591e11e/kube-rbac-proxy/0.log" Jul 17 18:26:09.498904 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:09.498876 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f99569965-6w8tx_48125a20-2cbd-4d74-a8c1-7945053655e9/thanos-query/0.log" Jul 17 18:26:09.520477 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:09.520348 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f99569965-6w8tx_48125a20-2cbd-4d74-a8c1-7945053655e9/kube-rbac-proxy-web/0.log" Jul 17 18:26:09.547912 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:09.547881 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f99569965-6w8tx_48125a20-2cbd-4d74-a8c1-7945053655e9/kube-rbac-proxy/0.log" Jul 17 18:26:09.570613 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:09.570587 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f99569965-6w8tx_48125a20-2cbd-4d74-a8c1-7945053655e9/prom-label-proxy/0.log" Jul 17 18:26:09.594410 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:09.594378 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f99569965-6w8tx_48125a20-2cbd-4d74-a8c1-7945053655e9/kube-rbac-proxy-rules/0.log" Jul 17 18:26:09.618040 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:09.618014 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f99569965-6w8tx_48125a20-2cbd-4d74-a8c1-7945053655e9/kube-rbac-proxy-metrics/0.log" Jul 17 18:26:11.758661 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:11.758622 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n"] Jul 17 18:26:11.760129 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:11.760096 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f9d56dd-db1f-4fd7-98c9-efae0226fd4a" containerName="revoke-keys" Jul 17 18:26:11.768405 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:11.768329 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n" Jul 17 18:26:11.771107 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:11.770840 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n"] Jul 17 18:26:11.864969 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:11.864930 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b91a097-3d37-4f7f-ace2-e233481d3511-sys\") pod \"perf-node-gather-daemonset-z2j9n\" (UID: \"1b91a097-3d37-4f7f-ace2-e233481d3511\") " pod="openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n" Jul 17 18:26:11.865270 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:11.865252 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fddgw\" (UniqueName: \"kubernetes.io/projected/1b91a097-3d37-4f7f-ace2-e233481d3511-kube-api-access-fddgw\") pod \"perf-node-gather-daemonset-z2j9n\" (UID: \"1b91a097-3d37-4f7f-ace2-e233481d3511\") " pod="openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n" Jul 17 18:26:11.865539 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:11.865509 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1b91a097-3d37-4f7f-ace2-e233481d3511-proc\") pod \"perf-node-gather-daemonset-z2j9n\" (UID: \"1b91a097-3d37-4f7f-ace2-e233481d3511\") " pod="openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n" Jul 17 18:26:11.865648 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:11.865623 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b91a097-3d37-4f7f-ace2-e233481d3511-lib-modules\") pod \"perf-node-gather-daemonset-z2j9n\" (UID: \"1b91a097-3d37-4f7f-ace2-e233481d3511\") " pod="openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n" Jul 17 18:26:11.865841 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:11.865821 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1b91a097-3d37-4f7f-ace2-e233481d3511-podres\") pod \"perf-node-gather-daemonset-z2j9n\" (UID: \"1b91a097-3d37-4f7f-ace2-e233481d3511\") " pod="openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n" Jul 17 18:26:11.966766 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:11.966715 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b91a097-3d37-4f7f-ace2-e233481d3511-lib-modules\") pod \"perf-node-gather-daemonset-z2j9n\" (UID: \"1b91a097-3d37-4f7f-ace2-e233481d3511\") " pod="openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n" Jul 17 18:26:11.966973 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:11.966776 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1b91a097-3d37-4f7f-ace2-e233481d3511-podres\") pod \"perf-node-gather-daemonset-z2j9n\" (UID: \"1b91a097-3d37-4f7f-ace2-e233481d3511\") " pod="openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n" Jul 17 18:26:11.966973 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:11.966839 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b91a097-3d37-4f7f-ace2-e233481d3511-sys\") pod \"perf-node-gather-daemonset-z2j9n\" (UID: \"1b91a097-3d37-4f7f-ace2-e233481d3511\") " pod="openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n" Jul 17 18:26:11.966973 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:11.966870 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fddgw\" (UniqueName: \"kubernetes.io/projected/1b91a097-3d37-4f7f-ace2-e233481d3511-kube-api-access-fddgw\") pod \"perf-node-gather-daemonset-z2j9n\" (UID: \"1b91a097-3d37-4f7f-ace2-e233481d3511\") " pod="openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n" Jul 17 18:26:11.967156 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:11.967038 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1b91a097-3d37-4f7f-ace2-e233481d3511-proc\") pod \"perf-node-gather-daemonset-z2j9n\" (UID: \"1b91a097-3d37-4f7f-ace2-e233481d3511\") " pod="openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n" Jul 17 18:26:11.967242 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:11.967224 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1b91a097-3d37-4f7f-ace2-e233481d3511-proc\") pod \"perf-node-gather-daemonset-z2j9n\" (UID: \"1b91a097-3d37-4f7f-ace2-e233481d3511\") " pod="openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n" Jul 17 18:26:11.967421 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:11.967399 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b91a097-3d37-4f7f-ace2-e233481d3511-lib-modules\") pod \"perf-node-gather-daemonset-z2j9n\" (UID: \"1b91a097-3d37-4f7f-ace2-e233481d3511\") " pod="openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n" Jul 17 18:26:11.967510 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:11.967496 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1b91a097-3d37-4f7f-ace2-e233481d3511-podres\") pod \"perf-node-gather-daemonset-z2j9n\" (UID: \"1b91a097-3d37-4f7f-ace2-e233481d3511\") " pod="openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n" Jul 17 18:26:11.967571 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:11.967545 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b91a097-3d37-4f7f-ace2-e233481d3511-sys\") pod \"perf-node-gather-daemonset-z2j9n\" (UID: \"1b91a097-3d37-4f7f-ace2-e233481d3511\") " pod="openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n" Jul 17 18:26:11.976224 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:11.976188 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fddgw\" (UniqueName: \"kubernetes.io/projected/1b91a097-3d37-4f7f-ace2-e233481d3511-kube-api-access-fddgw\") pod \"perf-node-gather-daemonset-z2j9n\" (UID: \"1b91a097-3d37-4f7f-ace2-e233481d3511\") " pod="openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n" Jul 17 18:26:12.066828 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:12.066744 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7cc68cc5c9-gs95r_affdc3b9-a61c-4b68-9436-7f3846d48355/console/0.log" Jul 17 18:26:12.083349 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:12.083308 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n" Jul 17 18:26:12.105983 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:12.105948 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-b4dgk_b7866d5c-3eba-485c-9930-3b7823dc8cce/download-server/0.log" Jul 17 18:26:12.283173 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:12.283141 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n"] Jul 17 18:26:12.284954 ip-10-0-142-155 kubenswrapper[2574]: W0717 18:26:12.284928 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1b91a097_3d37_4f7f_ace2_e233481d3511.slice/crio-cd4512d5826e43040b50d8ffb46261729ba5d0e487a227b4479945f1a1367bfb WatchSource:0}: Error finding container cd4512d5826e43040b50d8ffb46261729ba5d0e487a227b4479945f1a1367bfb: Status 404 returned error can't find the container with id cd4512d5826e43040b50d8ffb46261729ba5d0e487a227b4479945f1a1367bfb Jul 17 18:26:12.724030 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:12.723992 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n" event={"ID":"1b91a097-3d37-4f7f-ace2-e233481d3511","Type":"ContainerStarted","Data":"d5098d4c63ee88dbf66dd000404b290f7c0ea1ed207b9805f2a06cccfcb9a618"} Jul 17 18:26:12.724247 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:12.724055 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n" event={"ID":"1b91a097-3d37-4f7f-ace2-e233481d3511","Type":"ContainerStarted","Data":"cd4512d5826e43040b50d8ffb46261729ba5d0e487a227b4479945f1a1367bfb"} Jul 17 18:26:12.724247 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:12.724086 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n" Jul 17 18:26:12.742728 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:12.742675 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n" podStartSLOduration=1.74265472 podStartE2EDuration="1.74265472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:26:12.740966023 +0000 UTC m=+2184.328918714" watchObservedRunningTime="2026-07-17 18:26:12.74265472 +0000 UTC m=+2184.330607439" Jul 17 18:26:13.718486 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:13.718456 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kpgqx_7df87b83-ee0d-4171-a51f-311f238b59f8/dns/0.log" Jul 17 18:26:13.739695 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:13.739667 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kpgqx_7df87b83-ee0d-4171-a51f-311f238b59f8/kube-rbac-proxy/0.log" Jul 17 18:26:13.895601 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:13.895569 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sfcxh_db8a7cf2-a598-4192-b51b-815e6ad11718/dns-node-resolver/0.log" Jul 17 18:26:14.529487 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:14.529459 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sbkl7_cd6b07f4-0170-4df9-bcca-1e0613fe7079/node-ca/0.log" Jul 17 18:26:15.574734 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:15.574704 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6799b9db4b-nhjjk_eb3196bd-adc3-43b7-a538-9575d48afa1a/kube-auth-proxy/0.log" Jul 17 18:26:15.684294 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:15.684256 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-xbqzn_c483e1a0-1ed1-4f87-a9c9-aa84e01a72f2/payload-pre-processing/0.log" Jul 17 18:26:16.401904 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:16.401875 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-whhtb_2ff37b9c-1e7e-4daf-9b55-6bcd4c5d0261/serve-healthcheck-canary/0.log" Jul 17 18:26:16.976428 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:16.976381 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-86lxj_b348a8ff-2cb1-4d03-83ce-cf149b3e1691/kube-rbac-proxy/0.log" Jul 17 18:26:16.998415 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:16.998387 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-86lxj_b348a8ff-2cb1-4d03-83ce-cf149b3e1691/exporter/0.log" Jul 17 18:26:17.020071 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:17.020039 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-86lxj_b348a8ff-2cb1-4d03-83ce-cf149b3e1691/extractor/0.log" Jul 17 18:26:18.743995 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:18.743958 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wdmc6/perf-node-gather-daemonset-z2j9n" Jul 17 18:26:19.449672 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:19.449643 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cc99fdf4f-bmtvp_7f8367df-920a-417e-8033-8bad870315c6/manager/0.log" Jul 17 18:26:19.473807 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:19.473774 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cc99fdf4f-bmtvp_7f8367df-920a-417e-8033-8bad870315c6/copy-manifests/0.log" Jul 17 18:26:19.979719 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:19.979689 2574 scope.go:117] "RemoveContainer" containerID="3253fc13db80124907026ff60212b08c27e446099ff78f01ed8aa907fd7240c9" Jul 17 18:26:20.788178 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:20.788137 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j" event={"ID":"3f213ae2-3c83-45be-a3c8-32314c9d5162","Type":"ContainerStarted","Data":"ec4f8c9fc80d5173fabb830c328ab48a69b628b9d54640f190db01f7d04b28e3"} Jul 17 18:26:20.806298 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:20.806230 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j" podStartSLOduration=20.80620474 podStartE2EDuration="20.80620474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:26:03.667691643 +0000 UTC m=+2175.255644333" watchObservedRunningTime="2026-07-17 18:26:20.80620474 +0000 UTC m=+2192.394157431" Jul 17 18:26:20.877475 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:20.877441 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6568db77d8-ldt7z_a7c57b39-9500-4bce-bda2-b5ab525d97d6/manager/0.log" Jul 17 18:26:20.943949 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:20.943909 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-fxhg4_bb8d98a3-eae2-4640-8a32-b56c4a9b702a/openshift-lws-operator/0.log" Jul 17 18:26:21.797750 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:21.797710 2574 generic.go:356] "Generic (PLEG): container finished" podID="3f213ae2-3c83-45be-a3c8-32314c9d5162" containerID="ec4f8c9fc80d5173fabb830c328ab48a69b628b9d54640f190db01f7d04b28e3" exitCode=6 Jul 17 18:26:21.798241 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:21.797872 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j" event={"ID":"3f213ae2-3c83-45be-a3c8-32314c9d5162","Type":"ContainerDied","Data":"ec4f8c9fc80d5173fabb830c328ab48a69b628b9d54640f190db01f7d04b28e3"} Jul 17 18:26:21.798241 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:21.797931 2574 scope.go:117] "RemoveContainer" containerID="3253fc13db80124907026ff60212b08c27e446099ff78f01ed8aa907fd7240c9" Jul 17 18:26:21.798378 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:21.798342 2574 scope.go:117] "RemoveContainer" containerID="ec4f8c9fc80d5173fabb830c328ab48a69b628b9d54640f190db01f7d04b28e3" Jul 17 18:26:21.798653 ip-10-0-142-155 kubenswrapper[2574]: E0717 18:26:21.798624 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j_odh-ai-gateway-infra(3f213ae2-3c83-45be-a3c8-32314c9d5162)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j" podUID="3f213ae2-3c83-45be-a3c8-32314c9d5162" Jul 17 18:26:21.817085 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:21.817049 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j"] Jul 17 18:26:22.615408 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:22.615377 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6799b9db4b-nhjjk_eb3196bd-adc3-43b7-a538-9575d48afa1a/kube-auth-proxy/0.log" Jul 17 18:26:22.686033 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:22.685978 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-xbqzn_c483e1a0-1ed1-4f87-a9c9-aa84e01a72f2/payload-pre-processing/0.log" Jul 17 18:26:22.998656 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:22.998622 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j" Jul 17 18:26:23.092551 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:23.092514 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f213ae2-3c83-45be-a3c8-32314c9d5162-maas-api-service-ca\") pod \"3f213ae2-3c83-45be-a3c8-32314c9d5162\" (UID: \"3f213ae2-3c83-45be-a3c8-32314c9d5162\") " Jul 17 18:26:23.092892 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:23.092867 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f213ae2-3c83-45be-a3c8-32314c9d5162-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f213ae2-3c83-45be-a3c8-32314c9d5162" (UID: "3f213ae2-3c83-45be-a3c8-32314c9d5162"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:26:23.194394 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:23.194293 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f213ae2-3c83-45be-a3c8-32314c9d5162-maas-api-service-ca\") on node \"ip-10-0-142-155.ec2.internal\" DevicePath \"\"" Jul 17 18:26:23.815001 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:23.814907 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j" event={"ID":"3f213ae2-3c83-45be-a3c8-32314c9d5162","Type":"ContainerDied","Data":"d7ad4eb556724b37409bced37ff8d6df8ce62cd48b1a0d0abde9c7d9c4cd17e2"} Jul 17 18:26:23.815262 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:23.815243 2574 scope.go:117] "RemoveContainer" containerID="ec4f8c9fc80d5173fabb830c328ab48a69b628b9d54640f190db01f7d04b28e3" Jul 17 18:26:23.815472 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:23.815168 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j" Jul 17 18:26:23.845455 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:23.845420 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j"] Jul 17 18:26:23.851291 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:23.851263 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-db09a830-pmf6j"] Jul 17 18:26:23.990202 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:23.990171 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6799b9db4b-nhjjk_eb3196bd-adc3-43b7-a538-9575d48afa1a/kube-auth-proxy/0.log" Jul 17 18:26:24.073485 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:24.073381 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-xbqzn_c483e1a0-1ed1-4f87-a9c9-aa84e01a72f2/payload-pre-processing/0.log" Jul 17 18:26:24.989566 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:24.989504 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f213ae2-3c83-45be-a3c8-32314c9d5162" path="/var/lib/kubelet/pods/3f213ae2-3c83-45be-a3c8-32314c9d5162/volumes" Jul 17 18:26:27.992076 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:27.992046 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ncm4s_f044d914-e059-4c54-9ab3-b933c612b30a/kube-multus-additional-cni-plugins/0.log" Jul 17 18:26:28.019621 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:28.019586 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ncm4s_f044d914-e059-4c54-9ab3-b933c612b30a/egress-router-binary-copy/0.log" Jul 17 18:26:28.041605 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:28.041578 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ncm4s_f044d914-e059-4c54-9ab3-b933c612b30a/cni-plugins/0.log" Jul 17 18:26:28.066206 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:28.066165 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ncm4s_f044d914-e059-4c54-9ab3-b933c612b30a/bond-cni-plugin/0.log" Jul 17 18:26:28.091403 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:28.091348 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ncm4s_f044d914-e059-4c54-9ab3-b933c612b30a/routeoverride-cni/0.log" Jul 17 18:26:28.115341 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:28.115310 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ncm4s_f044d914-e059-4c54-9ab3-b933c612b30a/whereabouts-cni-bincopy/0.log" Jul 17 18:26:28.139756 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:28.139737 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ncm4s_f044d914-e059-4c54-9ab3-b933c612b30a/whereabouts-cni/0.log" Jul 17 18:26:28.411172 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:28.411078 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hzh2t_c0ed2480-e1bb-4002-bb74-b79be477ba8a/kube-multus/0.log" Jul 17 18:26:28.515768 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:28.515740 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r8hm5_315910cc-963f-4190-98a8-4453798d7fc7/network-metrics-daemon/0.log" Jul 17 18:26:28.532383 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:28.532331 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r8hm5_315910cc-963f-4190-98a8-4453798d7fc7/kube-rbac-proxy/0.log" Jul 17 18:26:29.542234 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:29.542197 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddl85_55537b77-ff5d-44be-b7f7-8001ab4dd01d/ovn-controller/0.log" Jul 17 18:26:29.557704 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:29.557671 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddl85_55537b77-ff5d-44be-b7f7-8001ab4dd01d/ovn-acl-logging/0.log" Jul 17 18:26:29.579737 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:29.579711 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddl85_55537b77-ff5d-44be-b7f7-8001ab4dd01d/kube-rbac-proxy-node/0.log" Jul 17 18:26:29.601608 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:29.601573 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddl85_55537b77-ff5d-44be-b7f7-8001ab4dd01d/kube-rbac-proxy-ovn-metrics/0.log" Jul 17 18:26:29.622895 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:29.622868 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddl85_55537b77-ff5d-44be-b7f7-8001ab4dd01d/northd/0.log" Jul 17 18:26:29.643241 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:29.643209 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddl85_55537b77-ff5d-44be-b7f7-8001ab4dd01d/nbdb/0.log" Jul 17 18:26:29.665517 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:29.665487 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddl85_55537b77-ff5d-44be-b7f7-8001ab4dd01d/sbdb/0.log" Jul 17 18:26:29.741082 ip-10-0-142-155 kubenswrapper[2574]: I0717 18:26:29.741046 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ddl85_55537b77-ff5d-44be-b7f7-8001ab4dd01d/ovnkube-controller/0.log"