Jul 17 12:22:35.739239 ip-10-0-134-119 systemd[1]: Starting Kubernetes Kubelet...
Jul 17 12:22:36.173347 ip-10-0-134-119 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 12:22:36.173347 ip-10-0-134-119 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 17 12:22:36.173347 ip-10-0-134-119 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 12:22:36.173347 ip-10-0-134-119 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 17 12:22:36.173347 ip-10-0-134-119 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 12:22:36.174318 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.174243    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 17 12:22:36.179500 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179481    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 12:22:36.179500 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179496    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 12:22:36.179500 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179501    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 12:22:36.179500 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179506    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 12:22:36.179733 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179510    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 12:22:36.179733 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179515    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 12:22:36.179733 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179521    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 12:22:36.179733 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179526    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 12:22:36.179733 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179531    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 12:22:36.179733 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179535    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 12:22:36.179733 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179539    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 12:22:36.179733 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179543    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 12:22:36.179733 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179547    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 12:22:36.179733 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179550    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 12:22:36.179733 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179554    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 12:22:36.179733 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179557    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 12:22:36.179733 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179561    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 12:22:36.179733 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179565    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 12:22:36.179733 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179569    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 12:22:36.179733 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179573    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 12:22:36.179733 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179577    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 12:22:36.179733 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179581    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 12:22:36.179733 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179585    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 12:22:36.179733 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179597    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 12:22:36.180628 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179601    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 12:22:36.180628 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179605    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 12:22:36.180628 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179609    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 12:22:36.180628 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179613    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 12:22:36.180628 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179617    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 12:22:36.180628 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179622    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 12:22:36.180628 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179626    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 12:22:36.180628 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179630    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 12:22:36.180628 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179635    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 12:22:36.180628 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179639    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 12:22:36.180628 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179643    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 12:22:36.180628 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179647    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 12:22:36.180628 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179651    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 12:22:36.180628 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179655    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 12:22:36.180628 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179660    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 12:22:36.180628 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179664    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 12:22:36.180628 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179668    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 12:22:36.180628 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179674    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 12:22:36.180628 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179680    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 12:22:36.181443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179686    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 12:22:36.181443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179691    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 12:22:36.181443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179695    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 12:22:36.181443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179700    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 12:22:36.181443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179705    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 12:22:36.181443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179710    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 12:22:36.181443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179715    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 12:22:36.181443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179719    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 12:22:36.181443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179724    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 12:22:36.181443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179728    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 12:22:36.181443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179732    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 12:22:36.181443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179740    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 12:22:36.181443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179747    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 12:22:36.181443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179751    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 12:22:36.181443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179755    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 12:22:36.181443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179760    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 12:22:36.181443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179764    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 12:22:36.181443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179769    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 12:22:36.181443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179773    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 12:22:36.181443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179777    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 12:22:36.182312 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179782    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 12:22:36.182312 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179786    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 12:22:36.182312 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179790    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 12:22:36.182312 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179794    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 12:22:36.182312 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179798    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 12:22:36.182312 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179803    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 12:22:36.182312 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179808    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 12:22:36.182312 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179812    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 12:22:36.182312 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179817    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 12:22:36.182312 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179821    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 12:22:36.182312 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179825    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 12:22:36.182312 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179830    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 12:22:36.182312 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179835    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 12:22:36.182312 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179839    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 12:22:36.182312 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179845    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 12:22:36.182312 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179849    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 12:22:36.182312 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179854    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 12:22:36.182312 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179858    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 12:22:36.182312 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179862    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 12:22:36.182312 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179866    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 12:22:36.183182 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179870    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 12:22:36.183182 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179874    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 12:22:36.183182 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179879    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 12:22:36.183182 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179884    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 12:22:36.183182 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179888    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 12:22:36.183182 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179892    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 12:22:36.183182 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179896    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 12:22:36.183182 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179901    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 12:22:36.183182 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179904    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 12:22:36.183182 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179909    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 12:22:36.183182 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179913    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 12:22:36.183182 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179917    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 12:22:36.183182 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179922    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 12:22:36.183182 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179926    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 12:22:36.183182 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179930    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 12:22:36.183182 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.179934    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 12:22:36.183182 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181133    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 12:22:36.183182 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181142    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 12:22:36.183182 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181148    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 12:22:36.183182 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181152    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 12:22:36.184059 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181156    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 12:22:36.184059 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181160    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 12:22:36.184059 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181165    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 12:22:36.184059 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181169    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 12:22:36.184059 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181173    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 12:22:36.184059 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181180    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 12:22:36.184059 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181184    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 12:22:36.184059 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181188    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 12:22:36.184059 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181192    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 12:22:36.184059 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181196    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 12:22:36.184059 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181200    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 12:22:36.184059 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181205    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 12:22:36.184059 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181209    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 12:22:36.184059 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181213    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 12:22:36.184059 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181217    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 12:22:36.184059 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181221    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 12:22:36.184059 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181225    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 12:22:36.184059 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181229    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 12:22:36.184059 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181234    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 12:22:36.184059 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181238    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 12:22:36.184675 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181242    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 12:22:36.184675 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181247    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 12:22:36.184675 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181251    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 12:22:36.184675 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181256    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 12:22:36.184675 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181260    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 12:22:36.184675 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181264    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 12:22:36.184675 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181268    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 12:22:36.184675 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181272    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 12:22:36.184675 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181277    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 12:22:36.184675 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181281    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 12:22:36.184675 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181285    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 12:22:36.184675 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181289    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 12:22:36.184675 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181293    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 12:22:36.184675 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181298    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 12:22:36.184675 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181305    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 12:22:36.184675 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181311    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 12:22:36.184675 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181316    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 12:22:36.184675 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181320    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 12:22:36.184675 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181328    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 12:22:36.185238 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181333    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 12:22:36.185238 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181337    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 12:22:36.185238 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181342    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 12:22:36.185238 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181346    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 12:22:36.185238 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181350    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 12:22:36.185238 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181354    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 12:22:36.185238 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181359    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 12:22:36.185238 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181362    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 12:22:36.185238 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181366    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 12:22:36.185238 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181371    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 12:22:36.185238 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181375    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 12:22:36.185238 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181379    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 12:22:36.185238 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181383    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 12:22:36.185238 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181388    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 12:22:36.185238 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181392    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 12:22:36.185238 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181399    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 12:22:36.185238 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181404    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 12:22:36.185238 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181409    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 12:22:36.185238 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181414    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 12:22:36.185238 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181419    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 12:22:36.185935 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181423    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 12:22:36.185935 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181428    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 12:22:36.185935 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181432    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 12:22:36.185935 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181436    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 12:22:36.185935 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181439    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 12:22:36.185935 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181443    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 12:22:36.185935 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181449    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 12:22:36.185935 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181454    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 12:22:36.185935 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181458    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 12:22:36.185935 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181462    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 12:22:36.185935 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181466    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 12:22:36.185935 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181470    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 12:22:36.185935 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181475    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 12:22:36.185935 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181481    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 12:22:36.185935 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181485    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 12:22:36.185935 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181489    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 12:22:36.185935 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181494    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 12:22:36.185935 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181498    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 12:22:36.185935 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181502    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 12:22:36.186443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181507    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 12:22:36.186443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181510    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 12:22:36.186443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181514    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 12:22:36.186443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181519    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 12:22:36.186443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181523    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 12:22:36.186443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181527    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 12:22:36.186443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181531    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 12:22:36.186443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181535    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 12:22:36.186443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181539    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 12:22:36.186443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181544    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 12:22:36.186443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181548    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 12:22:36.186443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181552    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 12:22:36.186443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181556    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 12:22:36.186443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181560    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 12:22:36.186443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181564    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 12:22:36.186443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181569    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 12:22:36.186443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.181574    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 12:22:36.186443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182145    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 12:22:36.186443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182154    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 12:22:36.186443 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182159    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 12:22:36.187039 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182163    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 12:22:36.187039 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182167    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 12:22:36.187039 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182171    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 12:22:36.187039 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182176    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 12:22:36.187039 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182180    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 12:22:36.187039 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182185    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 12:22:36.187039 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182189    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 12:22:36.187039 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182194    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 12:22:36.187039 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182198    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 12:22:36.187039 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182202    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 12:22:36.187039 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182206    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 12:22:36.187039 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182210    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 12:22:36.187039 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182215    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 12:22:36.187039 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182219    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 12:22:36.187039 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182223    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 12:22:36.187039 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182227    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 12:22:36.187039 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182231    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 12:22:36.187039 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182235    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 12:22:36.187039 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182239    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 12:22:36.187489 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182244    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 12:22:36.187489 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182248    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 12:22:36.187489 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182252    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 12:22:36.187489 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182256    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 12:22:36.187489 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182260    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 12:22:36.187489 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182264    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 12:22:36.187489 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182268    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 12:22:36.187489 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182272    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 12:22:36.187489 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182276    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 12:22:36.187489 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182281    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 12:22:36.187489 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182285    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 12:22:36.187489 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182289    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 12:22:36.187489 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182293    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 12:22:36.187489 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182299    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 12:22:36.187489 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182304    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 12:22:36.187489 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182311    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 12:22:36.187489 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182316    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 12:22:36.187489 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182321    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 12:22:36.187489 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182325    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 12:22:36.188008 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182330    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 12:22:36.188008 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182335    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 12:22:36.188008 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182339    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 12:22:36.188008 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182344    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 12:22:36.188008 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182348    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 12:22:36.188008 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182352    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 12:22:36.188008 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182357    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 12:22:36.188008 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182362    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 12:22:36.188008 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182366    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 12:22:36.188008 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182371    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 12:22:36.188008 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182376    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 12:22:36.188008 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182380    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 12:22:36.188008 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182384    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 12:22:36.188008 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182389    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 12:22:36.188008 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182393    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 12:22:36.188008 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182397    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 12:22:36.188008 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182401    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 12:22:36.188008 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182406    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 12:22:36.188008 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182410    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 12:22:36.188008 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182414    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 12:22:36.188492 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182419    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 12:22:36.188492 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182422    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 12:22:36.188492 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182427    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 12:22:36.188492 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182430    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 12:22:36.188492 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182435    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 12:22:36.188492 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182439    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 12:22:36.188492 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182443    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 12:22:36.188492 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182447    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 12:22:36.188492 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182451    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 12:22:36.188492 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182455    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 12:22:36.188492 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182459    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 12:22:36.188492 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182467    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 12:22:36.188492 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182472    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 12:22:36.188492 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182477    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 12:22:36.188492 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182482    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 12:22:36.188492 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182489    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 12:22:36.188492 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182494    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 12:22:36.188492 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182498    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 12:22:36.188492 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182502    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 12:22:36.188492 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182507    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 12:22:36.189025 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182511    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 12:22:36.189025 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182515    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 12:22:36.189025 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182519    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 12:22:36.189025 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182523    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 12:22:36.189025 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182527    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 12:22:36.189025 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182531    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 12:22:36.189025 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182535    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 12:22:36.189025 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182539    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 12:22:36.189025 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182543    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 12:22:36.189025 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182547    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 12:22:36.189025 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182551    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 12:22:36.189025 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182556    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 12:22:36.189025 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182560    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 12:22:36.189025 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182564    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 12:22:36.189025 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182568    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 12:22:36.189025 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182572    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 12:22:36.189025 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182576    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 12:22:36.189025 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.182580    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 12:22:36.189025 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182678    2565 flags.go:64] FLAG: --address="0.0.0.0"
Jul 17 12:22:36.189025 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182688    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 17 12:22:36.189025 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182696    2565 flags.go:64] FLAG: --anonymous-auth="true"
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182702    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182708    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182713    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182719    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182725    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182730    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182741    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182746    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182751    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182756    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182761    2565 flags.go:64] FLAG: --cgroup-root=""
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182765    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182770    2565 flags.go:64] FLAG: --client-ca-file=""
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182774    2565 flags.go:64] FLAG: --cloud-provider="external"
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182778    2565 flags.go:64] FLAG: --cluster-dns="[]"
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182785    2565 flags.go:64] FLAG: --cluster-domain=""
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182790    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182795    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182800    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182805    2565 flags.go:64] FLAG: --container-log-max-files="5"
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182811    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182816    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 17 12:22:36.189548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182821    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182826    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182831    2565 flags.go:64] FLAG: --contention-profiling="false"
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182836    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182840    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182845    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182849    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182855    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182860    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182864    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182869    2565 flags.go:64] FLAG: --enable-load-reader="false"
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182873    2565 flags.go:64] FLAG: --enable-server="true"
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182878    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182884    2565 flags.go:64] FLAG: --event-burst="100"
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182889    2565 flags.go:64] FLAG: --event-qps="50"
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182894    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182900    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182905    2565 flags.go:64] FLAG: --eviction-hard=""
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182911    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182915    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182920    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182925    2565 flags.go:64] FLAG: --eviction-soft=""
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182931    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182935    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182940    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 17 12:22:36.190172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182944    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182948    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182953    2565 flags.go:64] FLAG: --fail-swap-on="true"
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182958    2565 flags.go:64] FLAG: --feature-gates=""
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182964    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182969    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.182973    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183001    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183007    2565 flags.go:64] FLAG: --healthz-port="10248"
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183012    2565 flags.go:64] FLAG: --help="false"
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183017    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-134-119.ec2.internal"
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183021    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183026    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183030    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183036    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183041    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183046    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183050    2565 flags.go:64] FLAG: --image-service-endpoint=""
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183054    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183059    2565 flags.go:64] FLAG: --kube-api-burst="100"
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183063    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183069    2565 flags.go:64] FLAG: --kube-api-qps="50"
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183073    2565 flags.go:64] FLAG: --kube-reserved=""
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183079    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 17 12:22:36.190779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183084    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183090    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183094    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183099    2565 flags.go:64] FLAG: --lock-file=""
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183103    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183109    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183114    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183122    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183127    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183131    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183136    2565 flags.go:64] FLAG: --logging-format="text"
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183140    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183146    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183150    2565 flags.go:64] FLAG: --manifest-url=""
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183155    2565 flags.go:64] FLAG: --manifest-url-header=""
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183161    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183166    2565 flags.go:64] FLAG: --max-open-files="1000000"
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183172    2565 flags.go:64] FLAG: --max-pods="110"
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183176    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183181    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183185    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183189    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183194    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183198    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183203    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 17 12:22:36.191386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183218    2565 flags.go:64] FLAG: --node-status-max-images="50"
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183222    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183227    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183232    2565 flags.go:64] FLAG: --pod-cidr=""
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183237    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183245    2565 flags.go:64] FLAG: --pod-manifest-path=""
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183251    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183256    2565 flags.go:64] FLAG: --pods-per-core="0"
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183261    2565 flags.go:64] FLAG: --port="10250"
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183266    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183270    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06387ba6d41edc71a"
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183275    2565 flags.go:64] FLAG: --qos-reserved=""
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183280    2565 flags.go:64] FLAG: --read-only-port="10255"
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183287    2565 flags.go:64] FLAG: --register-node="true"
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183291    2565 flags.go:64] FLAG: --register-with-taints=""
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183297    2565 flags.go:64] FLAG: --registry-burst="10"
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183301    2565 flags.go:64] FLAG: --registry-qps="5"
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183306    2565 flags.go:64] FLAG: --reserved-cpus=""
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183311    2565 flags.go:64] FLAG: --reserved-memory=""
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183316    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183321    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183325    2565 flags.go:64] FLAG: --rotate-certificates="false"
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183329    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183334    2565 flags.go:64] FLAG: --runonce="false"
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183338    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 17 12:22:36.192022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183343    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183348    2565 flags.go:64] FLAG: --seccomp-default="false"
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183352    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183357    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183361    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183366    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183371    2565 flags.go:64] FLAG: --storage-driver-password="root"
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183375    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183379    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183384    2565 flags.go:64] FLAG: --storage-driver-user="root"
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183389    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183393    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183398    2565 flags.go:64] FLAG: --system-cgroups=""
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183404    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183412    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183416    2565 flags.go:64] FLAG: --tls-cert-file=""
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183421    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183431    2565 flags.go:64] FLAG: --tls-min-version=""
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183436    2565 flags.go:64] FLAG: --tls-private-key-file=""
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183440    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183446    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183451    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183455    2565 flags.go:64] FLAG: --v="2"
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183461    2565 flags.go:64] FLAG: --version="false"
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183468    2565 flags.go:64] FLAG: --vmodule=""
Jul 17 12:22:36.192655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183473    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 17 12:22:36.193292 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.183479    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 17 12:22:36.193292 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183649    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 12:22:36.193292 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183655    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 12:22:36.193292 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183660    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 12:22:36.193292 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183665    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 12:22:36.193292 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183669    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 12:22:36.193292 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183673    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 12:22:36.193292 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183677    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 12:22:36.193292 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183681    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 12:22:36.193292 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183686    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 12:22:36.193292 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183691    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 12:22:36.193292 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183695    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 12:22:36.193292 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183699    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 12:22:36.193292 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183706    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 12:22:36.193292 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183712    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 12:22:36.193292 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183716    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 12:22:36.193292 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183721    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 12:22:36.193292 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183726    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 12:22:36.193292 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183730    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 12:22:36.193292 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183736    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 12:22:36.193776 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183740    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 12:22:36.193776 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183744    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 12:22:36.193776 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183748    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 12:22:36.193776 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183753    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 12:22:36.193776 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183758    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 12:22:36.193776 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183762    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 12:22:36.193776 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183767    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 12:22:36.193776 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183772    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 12:22:36.193776 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183776    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 12:22:36.193776 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183780    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 12:22:36.193776 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183784    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 12:22:36.193776 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183788    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 12:22:36.193776 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183793    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 12:22:36.193776 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183797    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 12:22:36.193776 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183801    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 12:22:36.193776 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183805    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 12:22:36.193776 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183809    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 12:22:36.193776 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183814    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 12:22:36.193776 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183818    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 12:22:36.194287 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183824    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 12:22:36.194287 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183829    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 12:22:36.194287 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183833    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 12:22:36.194287 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183838    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 12:22:36.194287 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183842    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 12:22:36.194287 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183846    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 12:22:36.194287 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183850    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 12:22:36.194287 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183854    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 12:22:36.194287 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183860    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 12:22:36.194287 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183864    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 12:22:36.194287 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183868    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 12:22:36.194287 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183872    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 12:22:36.194287 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183877    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 12:22:36.194287 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183882    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 12:22:36.194287 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183886    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 12:22:36.194287 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183890    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 12:22:36.194287 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183895    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 12:22:36.194287 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183900    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 12:22:36.194287 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183904    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 12:22:36.194287 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183909    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 12:22:36.194761 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183913    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 12:22:36.194761 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183917    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 12:22:36.194761 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183921    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 12:22:36.194761 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183925    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 12:22:36.194761 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183929    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 12:22:36.194761 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183934    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 12:22:36.194761 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183938    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 12:22:36.194761 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183942    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 12:22:36.194761 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183946    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 12:22:36.194761 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183950    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 12:22:36.194761 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183954    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 12:22:36.194761 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183959    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 12:22:36.194761 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183963    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 12:22:36.194761 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183967    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 12:22:36.194761 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183971    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 12:22:36.194761 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.183975    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 12:22:36.194761 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184010    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 12:22:36.194761 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184015    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 12:22:36.194761 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184019    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 12:22:36.194761 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184024    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 12:22:36.195254 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184030    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 12:22:36.195254 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184036    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 12:22:36.195254 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184041    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 12:22:36.195254 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184046    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 12:22:36.195254 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184052    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 12:22:36.195254 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184056    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 12:22:36.195254 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184060    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 12:22:36.195254 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184064    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 12:22:36.195254 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184069    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 12:22:36.195254 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184073    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 12:22:36.195254 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184078    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 12:22:36.195254 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184084    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 12:22:36.195254 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184088    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 12:22:36.195254 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184092    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 12:22:36.195254 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184096    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 12:22:36.195254 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184100    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 12:22:36.195254 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184104    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 12:22:36.195254 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184108    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 12:22:36.195254 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184113    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 12:22:36.195701 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184117    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 12:22:36.195701 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.184121    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 12:22:36.195701 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.184745    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 12:22:36.195701 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.191543    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 17 12:22:36.195701 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.191557    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 17 12:22:36.195701 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.191569    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 17 12:22:36.195701 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.191577    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 17 12:22:36.195701 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191640    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 12:22:36.195701 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191645    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 12:22:36.195701 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191649    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 12:22:36.195701 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191652    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 12:22:36.195701 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191655    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 12:22:36.195701 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191658    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 12:22:36.195701 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191661    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 12:22:36.196068 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191664    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 12:22:36.196068 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191667    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 12:22:36.196068 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191670    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 12:22:36.196068 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191672    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 12:22:36.196068 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191674    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 12:22:36.196068 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191677    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 12:22:36.196068 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191680    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 12:22:36.196068 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191682    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 12:22:36.196068 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191685    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 12:22:36.196068 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191693    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 12:22:36.196068 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191696    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 12:22:36.196068 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191698    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 12:22:36.196068 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191701    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 12:22:36.196068 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191703    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 12:22:36.196068 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191706    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 12:22:36.196068 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191708    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 12:22:36.196068 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191711    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 12:22:36.196068 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191715    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 12:22:36.196068 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191718    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 12:22:36.196068 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191721    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 12:22:36.196544 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191724    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 12:22:36.196544 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191728    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 12:22:36.196544 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191732    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 12:22:36.196544 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191735    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 12:22:36.196544 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191738    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 12:22:36.196544 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191740    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 12:22:36.196544 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191743    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 12:22:36.196544 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191745    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 12:22:36.196544 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191747    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 12:22:36.196544 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191750    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 12:22:36.196544 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191752    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 12:22:36.196544 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191754    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 12:22:36.196544 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191757    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 12:22:36.196544 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191759    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 12:22:36.196544 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191762    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 12:22:36.196544 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191768    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 12:22:36.196544 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191770    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 12:22:36.196544 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191773    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 12:22:36.196544 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191775    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 12:22:36.196544 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191777    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 12:22:36.197133 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191780    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 12:22:36.197133 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191782    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 12:22:36.197133 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191791    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 12:22:36.197133 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191794    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 12:22:36.197133 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191796    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 12:22:36.197133 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191798    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 12:22:36.197133 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191801    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 12:22:36.197133 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191803    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 12:22:36.197133 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191806    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 12:22:36.197133 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191808    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 12:22:36.197133 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191811    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 12:22:36.197133 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191813    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 12:22:36.197133 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191815    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 12:22:36.197133 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191818    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 12:22:36.197133 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191821    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 12:22:36.197133 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191823    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 12:22:36.197133 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191826    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 12:22:36.197133 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191828    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 12:22:36.197133 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191830    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 12:22:36.197585 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191833    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 12:22:36.197585 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191835    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 12:22:36.197585 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191837    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 12:22:36.197585 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191839    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 12:22:36.197585 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191843    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 12:22:36.197585 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191846    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 12:22:36.197585 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191849    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 12:22:36.197585 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191852    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 12:22:36.197585 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191854    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 12:22:36.197585 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191857    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 12:22:36.197585 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191859    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 12:22:36.197585 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191861    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 12:22:36.197585 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191864    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 12:22:36.197585 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191866    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 12:22:36.197585 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191869    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 12:22:36.197585 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191871    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 12:22:36.197585 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191882    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 12:22:36.197585 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191885    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 12:22:36.197585 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191887    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 12:22:36.197585 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191890    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 12:22:36.198108 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191892    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 12:22:36.198108 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191894    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 12:22:36.198108 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191897    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 12:22:36.198108 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191899    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 12:22:36.198108 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191901    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 12:22:36.198108 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191904    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 12:22:36.198108 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191906    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 12:22:36.198108 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191909    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 12:22:36.198108 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191912    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 12:22:36.198108 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191914    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 12:22:36.198108 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191916    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 12:22:36.198108 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191918    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 12:22:36.198108 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.191921    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 12:22:36.198108 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.191926    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 12:22:36.198447 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192067    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 12:22:36.198447 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192072    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 12:22:36.198447 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192075    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 12:22:36.198447 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192078    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 12:22:36.198447 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192081    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 12:22:36.198447 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192083    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 12:22:36.198447 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192086    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 12:22:36.198447 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192089    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 12:22:36.198447 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192093    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 12:22:36.198447 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192096    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 12:22:36.198447 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192099    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 12:22:36.198447 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192101    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 12:22:36.198447 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192104    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 12:22:36.198447 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192107    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 12:22:36.198447 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192109    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 12:22:36.198447 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192118    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 12:22:36.198447 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192120    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 12:22:36.198447 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192123    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 12:22:36.198447 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192125    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 12:22:36.198903 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192128    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 12:22:36.198903 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192130    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 12:22:36.198903 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192132    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 12:22:36.198903 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192135    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 12:22:36.198903 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192139    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 12:22:36.198903 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192142    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 12:22:36.198903 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192144    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 12:22:36.198903 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192147    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 12:22:36.198903 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192149    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 12:22:36.198903 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192152    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 12:22:36.198903 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192154    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 12:22:36.198903 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192156    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 12:22:36.198903 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192158    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 12:22:36.198903 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192161    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 12:22:36.198903 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192163    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 12:22:36.198903 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192165    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 12:22:36.198903 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192168    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 12:22:36.198903 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192170    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 12:22:36.198903 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192172    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 12:22:36.199368 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192175    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 12:22:36.199368 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192177    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 12:22:36.199368 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192185    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 12:22:36.199368 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192187    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 12:22:36.199368 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192189    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 12:22:36.199368 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192192    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 12:22:36.199368 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192194    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 12:22:36.199368 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192196    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 12:22:36.199368 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192199    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 12:22:36.199368 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192201    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 12:22:36.199368 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192209    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 12:22:36.199368 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192211    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 12:22:36.199368 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192213    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 12:22:36.199368 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192216    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 12:22:36.199368 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192218    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 12:22:36.199368 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192220    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 12:22:36.199368 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192223    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 12:22:36.199368 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192225    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 12:22:36.199368 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192227    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 12:22:36.199368 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192230    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 12:22:36.199853 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192232    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 12:22:36.199853 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192234    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 12:22:36.199853 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192236    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 12:22:36.199853 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192239    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 12:22:36.199853 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192241    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 12:22:36.199853 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192243    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 12:22:36.199853 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192245    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 12:22:36.199853 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192248    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 12:22:36.199853 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192250    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 12:22:36.199853 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192252    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 12:22:36.199853 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192255    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 12:22:36.199853 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192257    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 12:22:36.199853 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192259    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 12:22:36.199853 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192262    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 12:22:36.199853 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192264    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 12:22:36.199853 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192267    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 12:22:36.199853 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192269    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 12:22:36.199853 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192271    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 12:22:36.199853 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192274    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 12:22:36.199853 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192276    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 12:22:36.200342 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192278    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 12:22:36.200342 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192281    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 12:22:36.200342 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192283    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 12:22:36.200342 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192291    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 12:22:36.200342 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192293    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 12:22:36.200342 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192296    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 12:22:36.200342 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192299    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 12:22:36.200342 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192303    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 12:22:36.200342 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192305    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 12:22:36.200342 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192308    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 12:22:36.200342 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192311    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 12:22:36.200342 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192313    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 12:22:36.200342 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192316    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 12:22:36.200342 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192318    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 12:22:36.200342 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192321    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 12:22:36.200342 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192324    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 12:22:36.200342 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192326    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 12:22:36.200342 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192328    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 12:22:36.200342 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192331    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 12:22:36.200938 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192333    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 12:22:36.200938 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:36.192335    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 12:22:36.200938 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.192340    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 12:22:36.200938 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.192442    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 17 12:22:36.200938 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.195563    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 17 12:22:36.200938 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.196955    2565 server.go:1019] "Starting client certificate rotation"
Jul 17 12:22:36.200938 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.197106    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 12:22:36.200938 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.197151    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 12:22:36.202623 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.202603    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 12:22:36.204452 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.204439    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 12:22:36.214186 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.214171    2565 log.go:25] "Validated CRI v1 runtime API"
Jul 17 12:22:36.217127 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.217114    2565 log.go:25] "Validated CRI v1 image API"
Jul 17 12:22:36.217878 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.217851    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 17 12:22:36.220507 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.220485    2565 fs.go:135] Filesystem UUIDs: map[777fb597-66a2-48e5-8458-7e42219cebb9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a1d90e9d-13e4-4698-8e22-9d9ed4195996:/dev/nvme0n1p4]
Jul 17 12:22:36.220554 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.220509    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 17 12:22:36.226445 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.226346    2565 manager.go:216] Machine: {Timestamp:2026-07-17 12:22:36.225175949 +0000 UTC m=+0.356560612 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3135146 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2494bbf77bd62b2a65ec5e1aeaf8ac SystemUUID:ec2494bb-f77b-d62b-2a65-ec5e1aeaf8ac BootID:4cbc6498-14a0-46b1-8b53-cbb3339197d6 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d7:9c:26:3d:b7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d7:9c:26:3d:b7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:2c:7e:e2:f9:0b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 17 12:22:36.226445 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.226442    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 17 12:22:36.226553 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.226504    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 17 12:22:36.230170 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.230149    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 17 12:22:36.230299 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.230174    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-119.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 17 12:22:36.230798 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.230788    2565 topology_manager.go:138] "Creating topology manager with none policy"
Jul 17 12:22:36.230827 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.230800    2565 container_manager_linux.go:309] "Creating device plugin manager"
Jul 17 12:22:36.230827 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.230808    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 12:22:36.230827 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.230820    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 12:22:36.231570 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.231556    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 12:22:36.232171 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.232160    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 17 12:22:36.234923 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.234912    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 12:22:36.235038 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.235029    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 17 12:22:36.235079 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.235056    2565 kubelet.go:486] "Attempting to sync node with API server"
Jul 17 12:22:36.235079 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.235072    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 17 12:22:36.235154 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.235146    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 17 12:22:36.235181 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.235170    2565 kubelet.go:398] "Adding apiserver pod source"
Jul 17 12:22:36.235212 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.235193    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 17 12:22:36.235740 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.235723    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 12:22:36.235798 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.235783    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 12:22:36.238226 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.238197    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 17 12:22:36.240896 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.240805    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 17 12:22:36.240973 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.240925    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 17 12:22:36.242402 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.242386    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 17 12:22:36.242474 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.242407    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 17 12:22:36.242474 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.242430    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 17 12:22:36.242474 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.242460    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 17 12:22:36.242474 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.242470    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 17 12:22:36.242474 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.242476    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 17 12:22:36.242612 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.242485    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 17 12:22:36.242612 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.242491    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 17 12:22:36.242612 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.242498    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 17 12:22:36.242612 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.242503    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 17 12:22:36.242612 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.242513    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 17 12:22:36.242612 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.242522    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 17 12:22:36.243334 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.243321    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 17 12:22:36.243334 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.243333    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 17 12:22:36.246636 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:36.246613    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 17 12:22:36.246636 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:36.246610    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-119.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 17 12:22:36.246730 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.246620    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-119.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 17 12:22:36.246857 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.246841    2565 server.go:1268] "Started kubelet"
Jul 17 12:22:36.246914 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.246888    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 17 12:22:36.247131 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.246999    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 17 12:22:36.247202 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.247154    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 17 12:22:36.247675 ip-10-0-134-119 systemd[1]: Started Kubernetes Kubelet.
Jul 17 12:22:36.248803 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.248789    2565 server.go:310] "Adding debug handlers to kubelet server"
Jul 17 12:22:36.248893 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.248835    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 17 12:22:36.252200 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:36.252170    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 17 12:22:36.253215 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:36.251996    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-119.ec2.internal.18c3132bcf087df9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-119.ec2.internal,UID:ip-10-0-134-119.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-119.ec2.internal,},FirstTimestamp:2026-07-17 12:22:36.246810105 +0000 UTC m=+0.378194756,LastTimestamp:2026-07-17 12:22:36.246810105 +0000 UTC m=+0.378194756,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-119.ec2.internal,}"
Jul 17 12:22:36.253812 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.253797    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vk96k"
Jul 17 12:22:36.255613 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.255593    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 17 12:22:36.255695 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.255598    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 17 12:22:36.255748 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:36.255718    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-119.ec2.internal\" not found"
Jul 17 12:22:36.256732 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.256703    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 17 12:22:36.256732 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.256703    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 17 12:22:36.256866 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.256741    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 17 12:22:36.256866 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.256790    2565 factory.go:55] Registering systemd factory
Jul 17 12:22:36.256866 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.256806    2565 factory.go:223] Registration of the systemd container factory successfully
Jul 17 12:22:36.256963 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.256878    2565 reconstruct.go:97] "Volume reconstruction finished"
Jul 17 12:22:36.256963 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.256887    2565 reconciler.go:29] "Reconciler: start to sync state"
Jul 17 12:22:36.257085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.257073    2565 factory.go:153] Registering CRI-O factory
Jul 17 12:22:36.257135 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.257088    2565 factory.go:223] Registration of the crio container factory successfully
Jul 17 12:22:36.257219 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.257136    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 17 12:22:36.257219 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.257158    2565 factory.go:103] Registering Raw factory
Jul 17 12:22:36.257219 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.257171    2565 manager.go:1196] Started watching for new ooms in manager
Jul 17 12:22:36.257766 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.257744    2565 manager.go:349] Starting recovery of all containers
Jul 17 12:22:36.259390 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.259366    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vk96k"
Jul 17 12:22:36.262001 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:36.261949    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 17 12:22:36.262102 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:36.262060    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-119.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 17 12:22:36.267384 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.267180    2565 manager.go:354] Recovery completed
Jul 17 12:22:36.270057 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:36.270035    2565 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 17 12:22:36.273359 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.273343    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 12:22:36.276185 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.276166    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-119.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 12:22:36.276268 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.276198    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-119.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 12:22:36.276268 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.276208    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-119.ec2.internal" event="NodeHasSufficientPID"
Jul 17 12:22:36.276733 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.276720    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 12:22:36.276733 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.276728    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 17 12:22:36.276846 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.276739    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 17 12:22:36.276846 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.276761    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 12:22:36.277421 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.277407    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-119.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 12:22:36.277505 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.277433    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-119.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 12:22:36.277505 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.277448    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-119.ec2.internal" event="NodeHasSufficientPID"
Jul 17 12:22:36.277911 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.277899    2565 policy_none.go:49] "None policy: Start"
Jul 17 12:22:36.277944 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.277916    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 17 12:22:36.277944 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.277925    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 17 12:22:36.278392 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:36.278328    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-119.ec2.internal.18c3132bd0c8ba02  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-119.ec2.internal,UID:ip-10-0-134-119.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-119.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-119.ec2.internal,},FirstTimestamp:2026-07-17 12:22:36.276185602 +0000 UTC m=+0.407570257,LastTimestamp:2026-07-17 12:22:36.276185602 +0000 UTC m=+0.407570257,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-119.ec2.internal,}"
Jul 17 12:22:36.279306 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.279294    2565 policy_none.go:47] "Start"
Jul 17 12:22:36.322500 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.322485    2565 manager.go:337] "Starting Device Plugin manager"
Jul 17 12:22:36.322585 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:36.322521    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 17 12:22:36.322585 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.322531    2565 server.go:87] "Starting device plugin registration server"
Jul 17 12:22:36.322771 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.322758    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 17 12:22:36.322825 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.322774    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 17 12:22:36.322898 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.322855    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 17 12:22:36.322946 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.322921    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 17 12:22:36.322946 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.322930    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 17 12:22:36.323381 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:36.323363    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 17 12:22:36.323452 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:36.323413    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-119.ec2.internal\" not found"
Jul 17 12:22:36.394672 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.394645    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 17 12:22:36.396182 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.395904    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 17 12:22:36.396182 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.395948    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 17 12:22:36.396182 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.395969    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 17 12:22:36.396182 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:36.396047    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 17 12:22:36.397917 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.397900    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 12:22:36.423873 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.423830    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 12:22:36.424574 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.424558    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-119.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 12:22:36.424658 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.424591    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-119.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 12:22:36.424658 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.424607    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-119.ec2.internal" event="NodeHasSufficientPID"
Jul 17 12:22:36.424658 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.424627    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-119.ec2.internal"
Jul 17 12:22:36.432457 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.432443    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-119.ec2.internal"
Jul 17 12:22:36.432498 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:36.432467    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-119.ec2.internal\": node \"ip-10-0-134-119.ec2.internal\" not found"
Jul 17 12:22:36.447263 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:36.447246    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-119.ec2.internal\" not found"
Jul 17 12:22:36.496976 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.496956    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-119.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-119.ec2.internal"]
Jul 17 12:22:36.497055 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.497036    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 12:22:36.498338 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.498326    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-119.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 12:22:36.498415 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.498353    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-119.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 12:22:36.498415 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.498367    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-119.ec2.internal" event="NodeHasSufficientPID"
Jul 17 12:22:36.499721 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.499708    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 12:22:36.499834 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.499822    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-119.ec2.internal"
Jul 17 12:22:36.499880 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.499851    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 12:22:36.500605 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.500584    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-119.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 12:22:36.500689 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.500608    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-119.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 12:22:36.500689 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.500620    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-119.ec2.internal" event="NodeHasSufficientPID"
Jul 17 12:22:36.500689 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.500658    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-119.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 12:22:36.500689 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.500676    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-119.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 12:22:36.500689 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.500685    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-119.ec2.internal" event="NodeHasSufficientPID"
Jul 17 12:22:36.501761 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.501745    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-119.ec2.internal"
Jul 17 12:22:36.501810 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.501777    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 12:22:36.502409 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.502390    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-119.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 12:22:36.502500 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.502426    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-119.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 12:22:36.502500 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.502442    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-119.ec2.internal" event="NodeHasSufficientPID"
Jul 17 12:22:36.529482 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:36.529459    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-119.ec2.internal\" not found" node="ip-10-0-134-119.ec2.internal"
Jul 17 12:22:36.533607 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:36.533592    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-119.ec2.internal\" not found" node="ip-10-0-134-119.ec2.internal"
Jul 17 12:22:36.547978 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:36.547962    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-119.ec2.internal\" not found"
Jul 17 12:22:36.558342 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.558324    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40802730aa52aba45795c84facc5eccc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-119.ec2.internal\" (UID: \"40802730aa52aba45795c84facc5eccc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-119.ec2.internal"
Jul 17 12:22:36.558431 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.558355    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40802730aa52aba45795c84facc5eccc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-119.ec2.internal\" (UID: \"40802730aa52aba45795c84facc5eccc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-119.ec2.internal"
Jul 17 12:22:36.558431 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.558380    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9e932a7529a4a2a40c2cedf7c627d21a-config\") pod \"kube-apiserver-proxy-ip-10-0-134-119.ec2.internal\" (UID: \"9e932a7529a4a2a40c2cedf7c627d21a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-119.ec2.internal"
Jul 17 12:22:36.649021 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:36.648978    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-119.ec2.internal\" not found"
Jul 17 12:22:36.659409 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.659373    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9e932a7529a4a2a40c2cedf7c627d21a-config\") pod \"kube-apiserver-proxy-ip-10-0-134-119.ec2.internal\" (UID: \"9e932a7529a4a2a40c2cedf7c627d21a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-119.ec2.internal"
Jul 17 12:22:36.659487 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.659420    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40802730aa52aba45795c84facc5eccc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-119.ec2.internal\" (UID: \"40802730aa52aba45795c84facc5eccc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-119.ec2.internal"
Jul 17 12:22:36.659487 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.659448    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40802730aa52aba45795c84facc5eccc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-119.ec2.internal\" (UID: \"40802730aa52aba45795c84facc5eccc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-119.ec2.internal"
Jul 17 12:22:36.659487 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.659472    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9e932a7529a4a2a40c2cedf7c627d21a-config\") pod \"kube-apiserver-proxy-ip-10-0-134-119.ec2.internal\" (UID: \"9e932a7529a4a2a40c2cedf7c627d21a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-119.ec2.internal"
Jul 17 12:22:36.659582 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.659484    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40802730aa52aba45795c84facc5eccc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-119.ec2.internal\" (UID: \"40802730aa52aba45795c84facc5eccc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-119.ec2.internal"
Jul 17 12:22:36.659582 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.659494    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40802730aa52aba45795c84facc5eccc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-119.ec2.internal\" (UID: \"40802730aa52aba45795c84facc5eccc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-119.ec2.internal"
Jul 17 12:22:36.749800 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:36.749751    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-119.ec2.internal\" not found"
Jul 17 12:22:36.832334 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.832299    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-119.ec2.internal"
Jul 17 12:22:36.836200 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:36.836061    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-119.ec2.internal"
Jul 17 12:22:36.850799 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:36.850774    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-119.ec2.internal\" not found"
Jul 17 12:22:36.951411 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:36.951370    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-119.ec2.internal\" not found"
Jul 17 12:22:37.051993 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:37.051920    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-119.ec2.internal\" not found"
Jul 17 12:22:37.152505 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:37.152478    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-119.ec2.internal\" not found"
Jul 17 12:22:37.182403 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.182368    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 12:22:37.196358 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.196329    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 17 12:22:37.196472 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.196454    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 12:22:37.196472 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.196458    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 12:22:37.235762 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.235734    2565 apiserver.go:52] "Watching apiserver"
Jul 17 12:22:37.248623 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.248598    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 12:22:37.256111 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.256085    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-119.ec2.internal"
Jul 17 12:22:37.256250 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.256150    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 17 12:22:37.258241 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.258215    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 17 12:22:37.261289 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.261266    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-bls5c","kube-system/konnectivity-agent-b6hrc","openshift-multus/multus-additional-cni-plugins-nb225","openshift-multus/network-metrics-daemon-ml8cr","openshift-network-diagnostics/network-check-target-v89v8","openshift-ovn-kubernetes/ovnkube-node-trg5s","kube-system/global-pull-secret-syncer-vqfdd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb","openshift-cluster-node-tuning-operator/tuned-m5cxt","openshift-dns/node-resolver-kw5vk","openshift-image-registry/node-ca-b25ws","openshift-multus/multus-sqgfk"]
Jul 17 12:22:37.263359 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.263341    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bls5c"
Jul 17 12:22:37.265594 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.265575    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b6hrc"
Jul 17 12:22:37.266591 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.266457    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 17 12:22:37.266591 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.266477    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 17 12:22:37.268162 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.266657    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dxvln\""
Jul 17 12:22:37.268162 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.266859    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 17 12:22:37.268265 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.268172    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 12:17:37 +0000 UTC" deadline="2028-04-08 19:31:00.241295296 +0000 UTC"
Jul 17 12:22:37.268265 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.268197    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15151h8m22.973102137s"
Jul 17 12:22:37.268682 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.268651    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 17 12:22:37.268768 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.268731    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 17 12:22:37.268768 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.268745    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cf6hz\""
Jul 17 12:22:37.268768 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.268747    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.270686 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.270666    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:22:37.270769 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:37.270741    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ml8cr" podUID="e5f413a7-be80-43f3-950e-be9b5c4bf9b4"
Jul 17 12:22:37.271514 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.271487    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 17 12:22:37.271832 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.271812    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 12:22:37.271949 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.271834    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 17 12:22:37.271949 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.271818    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 17 12:22:37.272118 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.272057    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:22:37.272175 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:37.272116    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v89v8" podUID="14587634-2d1d-472d-b470-981f1bc0ac8c"
Jul 17 12:22:37.272175 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.272057    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 17 12:22:37.272253 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.272177    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 17 12:22:37.272426 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.272412    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cpk4q\""
Jul 17 12:22:37.273425 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.273411    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 12:22:37.273490 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.273425    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.273534 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.273493    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-119.ec2.internal"
Jul 17 12:22:37.274971 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.274954    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:37.275114 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:37.275044    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfdd" podUID="881c9321-df8f-45af-a283-2fe1306b33b5"
Jul 17 12:22:37.276432 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.276410    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.277200 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.277180    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 17 12:22:37.277295 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.277190    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 17 12:22:37.277363 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.277257    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 17 12:22:37.277363 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.277261    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 17 12:22:37.277468 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.277373    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 17 12:22:37.277468 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.277379    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bpjc7\""
Jul 17 12:22:37.278368 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.278227    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.278775 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.278760    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 17 12:22:37.279006 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.278992    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 17 12:22:37.279615 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.279600    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 17 12:22:37.279708 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.279619    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 17 12:22:37.279708 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.279651    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2zvf4\""
Jul 17 12:22:37.279708 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.279701    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kw5vk"
Jul 17 12:22:37.280904 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.280880    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 17 12:22:37.281575 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.281319    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 17 12:22:37.281575 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.281365    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b25ws"
Jul 17 12:22:37.281575 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.281383    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qfspz\""
Jul 17 12:22:37.283037 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.283015    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-119.ec2.internal"]
Jul 17 12:22:37.283168 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.283152    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.286528 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.286509    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 17 12:22:37.286739 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.286725    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 17 12:22:37.286849 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.286834    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7xkpc\""
Jul 17 12:22:37.287412 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.287397    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-119.ec2.internal"]
Jul 17 12:22:37.287797 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.287785    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 12:22:37.287933 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.287920    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rfzz6\""
Jul 17 12:22:37.288172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.288159    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 17 12:22:37.288481 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.288468    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 17 12:22:37.289725 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.289713    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 17 12:22:37.289768 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.289734    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w54p6\""
Jul 17 12:22:37.290198 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.290183    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 17 12:22:37.315098 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.315075    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bcwt9"
Jul 17 12:22:37.331502 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.331478    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bcwt9"
Jul 17 12:22:37.357789 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.357762    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 17 12:22:37.363456 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.363433    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-host-run-ovn-kubernetes\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.363594 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.363473    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-os-release\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.363594 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.363538    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvtxh\" (UniqueName: \"kubernetes.io/projected/264b5b0f-2f3e-4acf-af1b-9d731a914efc-kube-api-access-lvtxh\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.363594 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.363585    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-log-socket\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.363779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.363621    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-host-run-netns\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.363779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.363647    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-host-var-lib-kubelet\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.363779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.363672    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f33412f3-025c-4c93-8301-3f3561d98055-device-dir\") pod \"aws-ebs-csi-driver-node-nmxkb\" (UID: \"f33412f3-025c-4c93-8301-3f3561d98055\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.363779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.363712    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-sys\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.363779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.363745    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-multus-socket-dir-parent\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.363779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.363773    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/264b5b0f-2f3e-4acf-af1b-9d731a914efc-system-cni-dir\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.364085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.363797    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f33412f3-025c-4c93-8301-3f3561d98055-etc-selinux\") pod \"aws-ebs-csi-driver-node-nmxkb\" (UID: \"f33412f3-025c-4c93-8301-3f3561d98055\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.364085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.363827    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7prh7\" (UniqueName: \"kubernetes.io/projected/f33412f3-025c-4c93-8301-3f3561d98055-kube-api-access-7prh7\") pod \"aws-ebs-csi-driver-node-nmxkb\" (UID: \"f33412f3-025c-4c93-8301-3f3561d98055\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.364085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.363851    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-host\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.364085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.363876    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-node-log\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.364085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.363903    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-ovn-node-metrics-cert\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.364085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.363935    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68g2g\" (UniqueName: \"kubernetes.io/projected/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-kube-api-access-68g2g\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.364085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.363957    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/881c9321-df8f-45af-a283-2fe1306b33b5-dbus\") pod \"global-pull-secret-syncer-vqfdd\" (UID: \"881c9321-df8f-45af-a283-2fe1306b33b5\") " pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:37.364085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.363982    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret\") pod \"global-pull-secret-syncer-vqfdd\" (UID: \"881c9321-df8f-45af-a283-2fe1306b33b5\") " pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:37.364085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364023    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-etc-sysctl-conf\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.364085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364039    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f33412f3-025c-4c93-8301-3f3561d98055-sys-fs\") pod \"aws-ebs-csi-driver-node-nmxkb\" (UID: \"f33412f3-025c-4c93-8301-3f3561d98055\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.364085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364061    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-run\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.364085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364077    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-etc-openvswitch\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.364085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364092    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-run-openvswitch\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.364629 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364107    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-host-cni-netd\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.364629 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364125    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-system-cni-dir\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.364629 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364145    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-hostroot\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.364629 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364160    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c97f4e9b-753e-4429-b477-03de51ee3fde-agent-certs\") pod \"konnectivity-agent-b6hrc\" (UID: \"c97f4e9b-753e-4429-b477-03de51ee3fde\") " pod="kube-system/konnectivity-agent-b6hrc"
Jul 17 12:22:37.364629 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364177    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/881c9321-df8f-45af-a283-2fe1306b33b5-kubelet-config\") pod \"global-pull-secret-syncer-vqfdd\" (UID: \"881c9321-df8f-45af-a283-2fe1306b33b5\") " pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:37.364629 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364191    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f33412f3-025c-4c93-8301-3f3561d98055-socket-dir\") pod \"aws-ebs-csi-driver-node-nmxkb\" (UID: \"f33412f3-025c-4c93-8301-3f3561d98055\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.364629 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364221    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de956422-3a00-4360-87d9-e43b2cb93859-host\") pod \"node-ca-b25ws\" (UID: \"de956422-3a00-4360-87d9-e43b2cb93859\") " pod="openshift-image-registry/node-ca-b25ws"
Jul 17 12:22:37.364629 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364251    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-run-ovn\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.364629 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364276    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-multus-conf-dir\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.364629 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364303    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/264b5b0f-2f3e-4acf-af1b-9d731a914efc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.364629 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364331    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f33412f3-025c-4c93-8301-3f3561d98055-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nmxkb\" (UID: \"f33412f3-025c-4c93-8301-3f3561d98055\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.364629 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364354    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-etc-sysctl-d\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.364629 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364377    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-multus-cni-dir\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.364629 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364428    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-host-var-lib-cni-multus\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.364629 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364455    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/264b5b0f-2f3e-4acf-af1b-9d731a914efc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.364629 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364473    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f33412f3-025c-4c93-8301-3f3561d98055-registration-dir\") pod \"aws-ebs-csi-driver-node-nmxkb\" (UID: \"f33412f3-025c-4c93-8301-3f3561d98055\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.365245 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364506    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-lib-modules\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.365245 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-var-lib-kubelet\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.365245 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364558    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcckr\" (UniqueName: \"kubernetes.io/projected/dd7bee8e-bd18-4de5-938c-a31fd97eb330-kube-api-access-rcckr\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.365245 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364586    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbpv8\" (UniqueName: \"kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8\") pod \"network-check-target-v89v8\" (UID: \"14587634-2d1d-472d-b470-981f1bc0ac8c\") " pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:22:37.365245 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364611    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd7bee8e-bd18-4de5-938c-a31fd97eb330-tmp\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.365245 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364634    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-systemd-units\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.365245 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364659    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-env-overrides\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.365245 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364699    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-host-run-multus-certs\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.365245 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364735    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/264b5b0f-2f3e-4acf-af1b-9d731a914efc-os-release\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.365245 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364765    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/264b5b0f-2f3e-4acf-af1b-9d731a914efc-cni-binary-copy\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.365245 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364791    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6nbf\" (UniqueName: \"kubernetes.io/projected/d0be7fb8-4dda-4b00-9b21-6c105e5bff69-kube-api-access-s6nbf\") pod \"node-resolver-kw5vk\" (UID: \"d0be7fb8-4dda-4b00-9b21-6c105e5bff69\") " pod="openshift-dns/node-resolver-kw5vk"
Jul 17 12:22:37.365245 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364821    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t55d\" (UniqueName: \"kubernetes.io/projected/eeee8e18-1099-4f80-b580-96a3d6af05a5-kube-api-access-5t55d\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.365245 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364848    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6w4c\" (UniqueName: \"kubernetes.io/projected/09613d5d-545c-4041-8019-b12539e97032-kube-api-access-b6w4c\") pod \"iptables-alerter-bls5c\" (UID: \"09613d5d-545c-4041-8019-b12539e97032\") " pod="openshift-network-operator/iptables-alerter-bls5c"
Jul 17 12:22:37.365245 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364876    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d229\" (UniqueName: \"kubernetes.io/projected/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-kube-api-access-9d229\") pod \"network-metrics-daemon-ml8cr\" (UID: \"e5f413a7-be80-43f3-950e-be9b5c4bf9b4\") " pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:22:37.365245 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364899    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-host-kubelet\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.365245 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364928    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-host-cni-bin\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.365808 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364960    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-host-run-k8s-cni-cncf-io\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.365808 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.364982    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c97f4e9b-753e-4429-b477-03de51ee3fde-konnectivity-ca\") pod \"konnectivity-agent-b6hrc\" (UID: \"c97f4e9b-753e-4429-b477-03de51ee3fde\") " pod="kube-system/konnectivity-agent-b6hrc"
Jul 17 12:22:37.365808 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365016    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-etc-sysconfig\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.365808 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365040    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-ovnkube-config\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.365808 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365055    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d0be7fb8-4dda-4b00-9b21-6c105e5bff69-hosts-file\") pod \"node-resolver-kw5vk\" (UID: \"d0be7fb8-4dda-4b00-9b21-6c105e5bff69\") " pod="openshift-dns/node-resolver-kw5vk"
Jul 17 12:22:37.365808 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365077    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/de956422-3a00-4360-87d9-e43b2cb93859-serviceca\") pod \"node-ca-b25ws\" (UID: \"de956422-3a00-4360-87d9-e43b2cb93859\") " pod="openshift-image-registry/node-ca-b25ws"
Jul 17 12:22:37.365808 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365113    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eeee8e18-1099-4f80-b580-96a3d6af05a5-cni-binary-copy\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.365808 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365135    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-etc-kubernetes\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.365808 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365176    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/09613d5d-545c-4041-8019-b12539e97032-iptables-alerter-script\") pod \"iptables-alerter-bls5c\" (UID: \"09613d5d-545c-4041-8019-b12539e97032\") " pod="openshift-network-operator/iptables-alerter-bls5c"
Jul 17 12:22:37.365808 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365207    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-etc-systemd\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.365808 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365227    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dd7bee8e-bd18-4de5-938c-a31fd97eb330-etc-tuned\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.365808 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365243    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-var-lib-openvswitch\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.365808 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365263    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-ovnkube-script-lib\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.365808 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365280    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg2ds\" (UniqueName: \"kubernetes.io/projected/de956422-3a00-4360-87d9-e43b2cb93859-kube-api-access-zg2ds\") pod \"node-ca-b25ws\" (UID: \"de956422-3a00-4360-87d9-e43b2cb93859\") " pod="openshift-image-registry/node-ca-b25ws"
Jul 17 12:22:37.365808 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365295    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eeee8e18-1099-4f80-b580-96a3d6af05a5-multus-daemon-config\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.365808 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365310    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/264b5b0f-2f3e-4acf-af1b-9d731a914efc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.366319 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365327    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs\") pod \"network-metrics-daemon-ml8cr\" (UID: \"e5f413a7-be80-43f3-950e-be9b5c4bf9b4\") " pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:22:37.366319 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365342    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-host-slash\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.366319 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365356    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-run-systemd\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.366319 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365372    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.366319 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365388    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0be7fb8-4dda-4b00-9b21-6c105e5bff69-tmp-dir\") pod \"node-resolver-kw5vk\" (UID: \"d0be7fb8-4dda-4b00-9b21-6c105e5bff69\") " pod="openshift-dns/node-resolver-kw5vk"
Jul 17 12:22:37.366319 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365403    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-cnibin\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.366319 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365418    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09613d5d-545c-4041-8019-b12539e97032-host-slash\") pod \"iptables-alerter-bls5c\" (UID: \"09613d5d-545c-4041-8019-b12539e97032\") " pod="openshift-network-operator/iptables-alerter-bls5c"
Jul 17 12:22:37.366319 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365441    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-etc-kubernetes\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.366319 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365461    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-host-run-netns\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.366319 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365480    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-host-var-lib-cni-bin\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.366319 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365504    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/264b5b0f-2f3e-4acf-af1b-9d731a914efc-cnibin\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.366319 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.365520    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-etc-modprobe-d\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.386944 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:37.386905    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40802730aa52aba45795c84facc5eccc.slice/crio-6c6df35b38b1541f44317f5ac46df7e8dd1308da9df8a04f9ab88fd0d1c3d594 WatchSource:0}: Error finding container 6c6df35b38b1541f44317f5ac46df7e8dd1308da9df8a04f9ab88fd0d1c3d594: Status 404 returned error can't find the container with id 6c6df35b38b1541f44317f5ac46df7e8dd1308da9df8a04f9ab88fd0d1c3d594
Jul 17 12:22:37.387440 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:37.387421    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e932a7529a4a2a40c2cedf7c627d21a.slice/crio-d3e4424d01fdc5d3ac2580bc4bca1831bd3cba7cff2514394ff57bea517e733f WatchSource:0}: Error finding container d3e4424d01fdc5d3ac2580bc4bca1831bd3cba7cff2514394ff57bea517e733f: Status 404 returned error can't find the container with id d3e4424d01fdc5d3ac2580bc4bca1831bd3cba7cff2514394ff57bea517e733f
Jul 17 12:22:37.392530 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.392513    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 12:22:37.399884 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.399837    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-119.ec2.internal" event={"ID":"9e932a7529a4a2a40c2cedf7c627d21a","Type":"ContainerStarted","Data":"d3e4424d01fdc5d3ac2580bc4bca1831bd3cba7cff2514394ff57bea517e733f"}
Jul 17 12:22:37.400975 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.400950    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-119.ec2.internal" event={"ID":"40802730aa52aba45795c84facc5eccc","Type":"ContainerStarted","Data":"6c6df35b38b1541f44317f5ac46df7e8dd1308da9df8a04f9ab88fd0d1c3d594"}
Jul 17 12:22:37.465863 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.465824    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-etc-openvswitch\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.466087 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.465904    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-run-openvswitch\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.466087 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.465931    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-host-cni-netd\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.466087 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.465939    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-etc-openvswitch\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.466087 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.465952    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-system-cni-dir\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.466087 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466008    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-hostroot\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.466087 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466028    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-run-openvswitch\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.466087 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466039    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-host-cni-netd\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.466087 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466055    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c97f4e9b-753e-4429-b477-03de51ee3fde-agent-certs\") pod \"konnectivity-agent-b6hrc\" (UID: \"c97f4e9b-753e-4429-b477-03de51ee3fde\") " pod="kube-system/konnectivity-agent-b6hrc"
Jul 17 12:22:37.466087 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466029    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-system-cni-dir\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.466087 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466068    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-hostroot\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.466598 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466102    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/881c9321-df8f-45af-a283-2fe1306b33b5-kubelet-config\") pod \"global-pull-secret-syncer-vqfdd\" (UID: \"881c9321-df8f-45af-a283-2fe1306b33b5\") " pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:37.466598 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466122    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f33412f3-025c-4c93-8301-3f3561d98055-socket-dir\") pod \"aws-ebs-csi-driver-node-nmxkb\" (UID: \"f33412f3-025c-4c93-8301-3f3561d98055\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.466598 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466143    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de956422-3a00-4360-87d9-e43b2cb93859-host\") pod \"node-ca-b25ws\" (UID: \"de956422-3a00-4360-87d9-e43b2cb93859\") " pod="openshift-image-registry/node-ca-b25ws"
Jul 17 12:22:37.466598 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-run-ovn\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.466598 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466178    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/881c9321-df8f-45af-a283-2fe1306b33b5-kubelet-config\") pod \"global-pull-secret-syncer-vqfdd\" (UID: \"881c9321-df8f-45af-a283-2fe1306b33b5\") " pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:37.466598 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466211    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-run-ovn\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.466598 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466210    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de956422-3a00-4360-87d9-e43b2cb93859-host\") pod \"node-ca-b25ws\" (UID: \"de956422-3a00-4360-87d9-e43b2cb93859\") " pod="openshift-image-registry/node-ca-b25ws"
Jul 17 12:22:37.466598 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466230    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-multus-conf-dir\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.466598 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466262    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-multus-conf-dir\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.466598 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466261    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/264b5b0f-2f3e-4acf-af1b-9d731a914efc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.466598 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466285    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f33412f3-025c-4c93-8301-3f3561d98055-socket-dir\") pod \"aws-ebs-csi-driver-node-nmxkb\" (UID: \"f33412f3-025c-4c93-8301-3f3561d98055\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.466598 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466298    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f33412f3-025c-4c93-8301-3f3561d98055-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nmxkb\" (UID: \"f33412f3-025c-4c93-8301-3f3561d98055\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.466598 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466324    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-etc-sysctl-d\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.466598 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466368    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f33412f3-025c-4c93-8301-3f3561d98055-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nmxkb\" (UID: \"f33412f3-025c-4c93-8301-3f3561d98055\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.466598 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466399    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-multus-cni-dir\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.466598 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466427    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-host-var-lib-cni-multus\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.466598 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/264b5b0f-2f3e-4acf-af1b-9d731a914efc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.467472 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466460    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-etc-sysctl-d\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.467472 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466481    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f33412f3-025c-4c93-8301-3f3561d98055-registration-dir\") pod \"aws-ebs-csi-driver-node-nmxkb\" (UID: \"f33412f3-025c-4c93-8301-3f3561d98055\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.467472 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466504    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-multus-cni-dir\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.467472 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466510    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-lib-modules\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.467472 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466493    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 17 12:22:37.467472 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466538    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-var-lib-kubelet\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.467472 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466561    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcckr\" (UniqueName: \"kubernetes.io/projected/dd7bee8e-bd18-4de5-938c-a31fd97eb330-kube-api-access-rcckr\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.467472 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466559    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f33412f3-025c-4c93-8301-3f3561d98055-registration-dir\") pod \"aws-ebs-csi-driver-node-nmxkb\" (UID: \"f33412f3-025c-4c93-8301-3f3561d98055\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.467472 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466586    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbpv8\" (UniqueName: \"kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8\") pod \"network-check-target-v89v8\" (UID: \"14587634-2d1d-472d-b470-981f1bc0ac8c\") " pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:22:37.467472 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466601    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-var-lib-kubelet\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.467472 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466641    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd7bee8e-bd18-4de5-938c-a31fd97eb330-tmp\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.467472 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466644    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-lib-modules\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.467472 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466663    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-systemd-units\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.467472 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466679    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-env-overrides\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.467472 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466705    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-host-run-multus-certs\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.467472 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466733    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/264b5b0f-2f3e-4acf-af1b-9d731a914efc-os-release\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.467472 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466751    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-systemd-units\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.467472 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466758    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/264b5b0f-2f3e-4acf-af1b-9d731a914efc-cni-binary-copy\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.468400 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466799    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6nbf\" (UniqueName: \"kubernetes.io/projected/d0be7fb8-4dda-4b00-9b21-6c105e5bff69-kube-api-access-s6nbf\") pod \"node-resolver-kw5vk\" (UID: \"d0be7fb8-4dda-4b00-9b21-6c105e5bff69\") " pod="openshift-dns/node-resolver-kw5vk"
Jul 17 12:22:37.468400 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466831    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t55d\" (UniqueName: \"kubernetes.io/projected/eeee8e18-1099-4f80-b580-96a3d6af05a5-kube-api-access-5t55d\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.468400 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466859    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6w4c\" (UniqueName: \"kubernetes.io/projected/09613d5d-545c-4041-8019-b12539e97032-kube-api-access-b6w4c\") pod \"iptables-alerter-bls5c\" (UID: \"09613d5d-545c-4041-8019-b12539e97032\") " pod="openshift-network-operator/iptables-alerter-bls5c"
Jul 17 12:22:37.468400 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466884    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d229\" (UniqueName: \"kubernetes.io/projected/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-kube-api-access-9d229\") pod \"network-metrics-daemon-ml8cr\" (UID: \"e5f413a7-be80-43f3-950e-be9b5c4bf9b4\") " pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:22:37.468400 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466913    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-host-kubelet\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.468400 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466937    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-host-cni-bin\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.468400 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466964    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-host-run-k8s-cni-cncf-io\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.468400 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466987    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/264b5b0f-2f3e-4acf-af1b-9d731a914efc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.468400 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466991    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c97f4e9b-753e-4429-b477-03de51ee3fde-konnectivity-ca\") pod \"konnectivity-agent-b6hrc\" (UID: \"c97f4e9b-753e-4429-b477-03de51ee3fde\") " pod="kube-system/konnectivity-agent-b6hrc"
Jul 17 12:22:37.468400 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467108    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-etc-sysconfig\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.468400 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467131    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-ovnkube-config\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.468400 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467153    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d0be7fb8-4dda-4b00-9b21-6c105e5bff69-hosts-file\") pod \"node-resolver-kw5vk\" (UID: \"d0be7fb8-4dda-4b00-9b21-6c105e5bff69\") " pod="openshift-dns/node-resolver-kw5vk"
Jul 17 12:22:37.468400 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467190    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/de956422-3a00-4360-87d9-e43b2cb93859-serviceca\") pod \"node-ca-b25ws\" (UID: \"de956422-3a00-4360-87d9-e43b2cb93859\") " pod="openshift-image-registry/node-ca-b25ws"
Jul 17 12:22:37.468400 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467216    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eeee8e18-1099-4f80-b580-96a3d6af05a5-cni-binary-copy\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.468400 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467254    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-etc-kubernetes\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.468400 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467280    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-env-overrides\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.468400 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467297    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/09613d5d-545c-4041-8019-b12539e97032-iptables-alerter-script\") pod \"iptables-alerter-bls5c\" (UID: \"09613d5d-545c-4041-8019-b12539e97032\") " pod="openshift-network-operator/iptables-alerter-bls5c"
Jul 17 12:22:37.469271 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467310    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/264b5b0f-2f3e-4acf-af1b-9d731a914efc-cni-binary-copy\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.469271 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467327    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-etc-systemd\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.469271 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467366    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-etc-kubernetes\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.469271 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467374    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dd7bee8e-bd18-4de5-938c-a31fd97eb330-etc-tuned\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.469271 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467400    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-var-lib-openvswitch\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.469271 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467410    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-host-run-multus-certs\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.469271 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467423    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-ovnkube-script-lib\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.469271 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467448    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg2ds\" (UniqueName: \"kubernetes.io/projected/de956422-3a00-4360-87d9-e43b2cb93859-kube-api-access-zg2ds\") pod \"node-ca-b25ws\" (UID: \"de956422-3a00-4360-87d9-e43b2cb93859\") " pod="openshift-image-registry/node-ca-b25ws"
Jul 17 12:22:37.469271 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467471    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/264b5b0f-2f3e-4acf-af1b-9d731a914efc-os-release\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.469271 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467479    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eeee8e18-1099-4f80-b580-96a3d6af05a5-multus-daemon-config\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.469271 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467314    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-etc-sysconfig\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.469271 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467520    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/264b5b0f-2f3e-4acf-af1b-9d731a914efc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.469271 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467551    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs\") pod \"network-metrics-daemon-ml8cr\" (UID: \"e5f413a7-be80-43f3-950e-be9b5c4bf9b4\") " pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:22:37.469271 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467570    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c97f4e9b-753e-4429-b477-03de51ee3fde-konnectivity-ca\") pod \"konnectivity-agent-b6hrc\" (UID: \"c97f4e9b-753e-4429-b477-03de51ee3fde\") " pod="kube-system/konnectivity-agent-b6hrc"
Jul 17 12:22:37.469271 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467578    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-host-slash\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.469271 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467615    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-run-systemd\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.469271 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467636    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-host-kubelet\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.470085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467681    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-host-cni-bin\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.470085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467732    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-host-run-k8s-cni-cncf-io\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.470085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467800    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d0be7fb8-4dda-4b00-9b21-6c105e5bff69-hosts-file\") pod \"node-resolver-kw5vk\" (UID: \"d0be7fb8-4dda-4b00-9b21-6c105e5bff69\") " pod="openshift-dns/node-resolver-kw5vk"
Jul 17 12:22:37.470085 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:37.467830    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:37.470085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467986    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/264b5b0f-2f3e-4acf-af1b-9d731a914efc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.470085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-ovnkube-config\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.470085 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:37.467995    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs podName:e5f413a7-be80-43f3-950e-be9b5c4bf9b4 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:37.967931713 +0000 UTC m=+2.099316445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs") pod "network-metrics-daemon-ml8cr" (UID: "e5f413a7-be80-43f3-950e-be9b5c4bf9b4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:37.470085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.466522    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-host-var-lib-cni-multus\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.470085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468083    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-host-slash\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.470085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468147    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/de956422-3a00-4360-87d9-e43b2cb93859-serviceca\") pod \"node-ca-b25ws\" (UID: \"de956422-3a00-4360-87d9-e43b2cb93859\") " pod="openshift-image-registry/node-ca-b25ws"
Jul 17 12:22:37.470085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.467647    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.470085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468218    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0be7fb8-4dda-4b00-9b21-6c105e5bff69-tmp-dir\") pod \"node-resolver-kw5vk\" (UID: \"d0be7fb8-4dda-4b00-9b21-6c105e5bff69\") " pod="openshift-dns/node-resolver-kw5vk"
Jul 17 12:22:37.470085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468224    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-var-lib-openvswitch\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.470085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468242    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-run-systemd\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.470085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468247    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-cnibin\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.470085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468275    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09613d5d-545c-4041-8019-b12539e97032-host-slash\") pod \"iptables-alerter-bls5c\" (UID: \"09613d5d-545c-4041-8019-b12539e97032\") " pod="openshift-network-operator/iptables-alerter-bls5c"
Jul 17 12:22:37.470085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468397    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-etc-kubernetes\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.470617 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468421    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-host-run-netns\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.470617 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468447    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-host-var-lib-cni-bin\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.470617 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468489    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/264b5b0f-2f3e-4acf-af1b-9d731a914efc-cnibin\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.470617 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468516    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-etc-modprobe-d\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.470617 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468541    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-host-run-ovn-kubernetes\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.470617 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468565    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-os-release\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.470617 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468605    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eeee8e18-1099-4f80-b580-96a3d6af05a5-cni-binary-copy\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.470617 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468593    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvtxh\" (UniqueName: \"kubernetes.io/projected/264b5b0f-2f3e-4acf-af1b-9d731a914efc-kube-api-access-lvtxh\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.470617 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468616    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0be7fb8-4dda-4b00-9b21-6c105e5bff69-tmp-dir\") pod \"node-resolver-kw5vk\" (UID: \"d0be7fb8-4dda-4b00-9b21-6c105e5bff69\") " pod="openshift-dns/node-resolver-kw5vk"
Jul 17 12:22:37.470617 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468637    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/264b5b0f-2f3e-4acf-af1b-9d731a914efc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.470617 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468649    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-log-socket\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.470617 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468677    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.470617 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468722    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-log-socket\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.470617 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468737    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-etc-kubernetes\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.470617 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468731    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-cnibin\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.470617 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468763    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-host-run-netns\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.470617 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468779    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-ovnkube-script-lib\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.471330 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468801    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-host-run-netns\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.471330 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468680    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-host-run-netns\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.471330 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468867    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09613d5d-545c-4041-8019-b12539e97032-host-slash\") pod \"iptables-alerter-bls5c\" (UID: \"09613d5d-545c-4041-8019-b12539e97032\") " pod="openshift-network-operator/iptables-alerter-bls5c"
Jul 17 12:22:37.471330 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468871    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-host-var-lib-kubelet\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.471330 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468908    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f33412f3-025c-4c93-8301-3f3561d98055-device-dir\") pod \"aws-ebs-csi-driver-node-nmxkb\" (UID: \"f33412f3-025c-4c93-8301-3f3561d98055\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.471330 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468916    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/264b5b0f-2f3e-4acf-af1b-9d731a914efc-cnibin\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.471330 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468956    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-host-var-lib-kubelet\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.471330 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.468970    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-etc-systemd\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.471330 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469003    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-etc-modprobe-d\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.471330 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469032    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f33412f3-025c-4c93-8301-3f3561d98055-device-dir\") pod \"aws-ebs-csi-driver-node-nmxkb\" (UID: \"f33412f3-025c-4c93-8301-3f3561d98055\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.471330 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469089    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-sys\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.471330 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469099    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-host-run-ovn-kubernetes\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.471330 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469193    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-multus-socket-dir-parent\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.471330 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469210    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-sys\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.471330 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469223    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/264b5b0f-2f3e-4acf-af1b-9d731a914efc-system-cni-dir\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.471330 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469249    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f33412f3-025c-4c93-8301-3f3561d98055-etc-selinux\") pod \"aws-ebs-csi-driver-node-nmxkb\" (UID: \"f33412f3-025c-4c93-8301-3f3561d98055\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.471330 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469275    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7prh7\" (UniqueName: \"kubernetes.io/projected/f33412f3-025c-4c93-8301-3f3561d98055-kube-api-access-7prh7\") pod \"aws-ebs-csi-driver-node-nmxkb\" (UID: \"f33412f3-025c-4c93-8301-3f3561d98055\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.472168 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469265    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-os-release\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.472168 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469303    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-host\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.472168 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469326    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-node-log\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.472168 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469335    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/264b5b0f-2f3e-4acf-af1b-9d731a914efc-system-cni-dir\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.472168 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469351    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-ovn-node-metrics-cert\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.472168 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469363    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eeee8e18-1099-4f80-b580-96a3d6af05a5-multus-daemon-config\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.472168 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469380    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68g2g\" (UniqueName: \"kubernetes.io/projected/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-kube-api-access-68g2g\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.472168 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469387    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-multus-socket-dir-parent\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.472168 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469423    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/881c9321-df8f-45af-a283-2fe1306b33b5-dbus\") pod \"global-pull-secret-syncer-vqfdd\" (UID: \"881c9321-df8f-45af-a283-2fe1306b33b5\") " pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:37.472168 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469427    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eeee8e18-1099-4f80-b580-96a3d6af05a5-host-var-lib-cni-bin\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.472168 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469455    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret\") pod \"global-pull-secret-syncer-vqfdd\" (UID: \"881c9321-df8f-45af-a283-2fe1306b33b5\") " pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:37.472168 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469447    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/09613d5d-545c-4041-8019-b12539e97032-iptables-alerter-script\") pod \"iptables-alerter-bls5c\" (UID: \"09613d5d-545c-4041-8019-b12539e97032\") " pod="openshift-network-operator/iptables-alerter-bls5c"
Jul 17 12:22:37.472168 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469496    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-node-log\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.472168 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469485    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-etc-sysctl-conf\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.472168 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469549    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-host\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.472168 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469538    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f33412f3-025c-4c93-8301-3f3561d98055-sys-fs\") pod \"aws-ebs-csi-driver-node-nmxkb\" (UID: \"f33412f3-025c-4c93-8301-3f3561d98055\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.472168 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469587    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f33412f3-025c-4c93-8301-3f3561d98055-etc-selinux\") pod \"aws-ebs-csi-driver-node-nmxkb\" (UID: \"f33412f3-025c-4c93-8301-3f3561d98055\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.472982 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469609    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f33412f3-025c-4c93-8301-3f3561d98055-sys-fs\") pod \"aws-ebs-csi-driver-node-nmxkb\" (UID: \"f33412f3-025c-4c93-8301-3f3561d98055\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.472982 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:37.469625    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:37.472982 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:37.469669    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret podName:881c9321-df8f-45af-a283-2fe1306b33b5 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:37.969654287 +0000 UTC m=+2.101038934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret") pod "global-pull-secret-syncer-vqfdd" (UID: "881c9321-df8f-45af-a283-2fe1306b33b5") : object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:37.472982 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469688    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/881c9321-df8f-45af-a283-2fe1306b33b5-dbus\") pod \"global-pull-secret-syncer-vqfdd\" (UID: \"881c9321-df8f-45af-a283-2fe1306b33b5\") " pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:37.472982 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469693    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-run\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.472982 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469714    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-etc-sysctl-conf\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.472982 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.469745    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dd7bee8e-bd18-4de5-938c-a31fd97eb330-run\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.472982 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.470428    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd7bee8e-bd18-4de5-938c-a31fd97eb330-tmp\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.472982 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.471731    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dd7bee8e-bd18-4de5-938c-a31fd97eb330-etc-tuned\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.472982 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.471863    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c97f4e9b-753e-4429-b477-03de51ee3fde-agent-certs\") pod \"konnectivity-agent-b6hrc\" (UID: \"c97f4e9b-753e-4429-b477-03de51ee3fde\") " pod="kube-system/konnectivity-agent-b6hrc"
Jul 17 12:22:37.472982 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.472054    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-ovn-node-metrics-cert\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.483217 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:37.483190    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 12:22:37.483217 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:37.483214    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 12:22:37.483380 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:37.483229    2565 projected.go:196] Error preparing data for projected volume kube-api-access-hbpv8 for pod openshift-network-diagnostics/network-check-target-v89v8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:37.483380 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:37.483291    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8 podName:14587634-2d1d-472d-b470-981f1bc0ac8c nodeName:}" failed. No retries permitted until 2026-07-17 12:22:37.983277788 +0000 UTC m=+2.114662450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hbpv8" (UniqueName: "kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8") pod "network-check-target-v89v8" (UID: "14587634-2d1d-472d-b470-981f1bc0ac8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:37.483624 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.483601    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6nbf\" (UniqueName: \"kubernetes.io/projected/d0be7fb8-4dda-4b00-9b21-6c105e5bff69-kube-api-access-s6nbf\") pod \"node-resolver-kw5vk\" (UID: \"d0be7fb8-4dda-4b00-9b21-6c105e5bff69\") " pod="openshift-dns/node-resolver-kw5vk"
Jul 17 12:22:37.483847 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.483826    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d229\" (UniqueName: \"kubernetes.io/projected/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-kube-api-access-9d229\") pod \"network-metrics-daemon-ml8cr\" (UID: \"e5f413a7-be80-43f3-950e-be9b5c4bf9b4\") " pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:22:37.485161 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.485129    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t55d\" (UniqueName: \"kubernetes.io/projected/eeee8e18-1099-4f80-b580-96a3d6af05a5-kube-api-access-5t55d\") pod \"multus-sqgfk\" (UID: \"eeee8e18-1099-4f80-b580-96a3d6af05a5\") " pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.485383 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.485358    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7prh7\" (UniqueName: \"kubernetes.io/projected/f33412f3-025c-4c93-8301-3f3561d98055-kube-api-access-7prh7\") pod \"aws-ebs-csi-driver-node-nmxkb\" (UID: \"f33412f3-025c-4c93-8301-3f3561d98055\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.486420 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.486401    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6w4c\" (UniqueName: \"kubernetes.io/projected/09613d5d-545c-4041-8019-b12539e97032-kube-api-access-b6w4c\") pod \"iptables-alerter-bls5c\" (UID: \"09613d5d-545c-4041-8019-b12539e97032\") " pod="openshift-network-operator/iptables-alerter-bls5c"
Jul 17 12:22:37.486597 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.486577    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68g2g\" (UniqueName: \"kubernetes.io/projected/ed8c8549-e57b-4306-b02b-7d1f466f7b4e-kube-api-access-68g2g\") pod \"ovnkube-node-trg5s\" (UID: \"ed8c8549-e57b-4306-b02b-7d1f466f7b4e\") " pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.487003 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.486987    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcckr\" (UniqueName: \"kubernetes.io/projected/dd7bee8e-bd18-4de5-938c-a31fd97eb330-kube-api-access-rcckr\") pod \"tuned-m5cxt\" (UID: \"dd7bee8e-bd18-4de5-938c-a31fd97eb330\") " pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.487615 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.487591    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvtxh\" (UniqueName: \"kubernetes.io/projected/264b5b0f-2f3e-4acf-af1b-9d731a914efc-kube-api-access-lvtxh\") pod \"multus-additional-cni-plugins-nb225\" (UID: \"264b5b0f-2f3e-4acf-af1b-9d731a914efc\") " pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.487678 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.487657    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg2ds\" (UniqueName: \"kubernetes.io/projected/de956422-3a00-4360-87d9-e43b2cb93859-kube-api-access-zg2ds\") pod \"node-ca-b25ws\" (UID: \"de956422-3a00-4360-87d9-e43b2cb93859\") " pod="openshift-image-registry/node-ca-b25ws"
Jul 17 12:22:37.595669 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.595567    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bls5c"
Jul 17 12:22:37.604521 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:37.604498    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09613d5d_545c_4041_8019_b12539e97032.slice/crio-1765cf6f2d2ddf3df60696c1d39b065f4994d51f7998a9520fda0590cea4ef34 WatchSource:0}: Error finding container 1765cf6f2d2ddf3df60696c1d39b065f4994d51f7998a9520fda0590cea4ef34: Status 404 returned error can't find the container with id 1765cf6f2d2ddf3df60696c1d39b065f4994d51f7998a9520fda0590cea4ef34
Jul 17 12:22:37.613659 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.613638    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b6hrc"
Jul 17 12:22:37.618779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.618762    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nb225"
Jul 17 12:22:37.621006 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:37.620984    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97f4e9b_753e_4429_b477_03de51ee3fde.slice/crio-9c9bcf93f995b119af7c0a0729c9d9e9b771921b36022cccf5eb2da0cc890519 WatchSource:0}: Error finding container 9c9bcf93f995b119af7c0a0729c9d9e9b771921b36022cccf5eb2da0cc890519: Status 404 returned error can't find the container with id 9c9bcf93f995b119af7c0a0729c9d9e9b771921b36022cccf5eb2da0cc890519
Jul 17 12:22:37.627152 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:37.627004    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264b5b0f_2f3e_4acf_af1b_9d731a914efc.slice/crio-f81a92798e33321e05f8ae9340512975759d09f652b83ddaf1b4bea41bb7309d WatchSource:0}: Error finding container f81a92798e33321e05f8ae9340512975759d09f652b83ddaf1b4bea41bb7309d: Status 404 returned error can't find the container with id f81a92798e33321e05f8ae9340512975759d09f652b83ddaf1b4bea41bb7309d
Jul 17 12:22:37.628224 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.628200    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:22:37.635723 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:37.635703    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8c8549_e57b_4306_b02b_7d1f466f7b4e.slice/crio-ac424194fef8797a931a21f456ac386b7ee40dac60b2213b4649544971308a7a WatchSource:0}: Error finding container ac424194fef8797a931a21f456ac386b7ee40dac60b2213b4649544971308a7a: Status 404 returned error can't find the container with id ac424194fef8797a931a21f456ac386b7ee40dac60b2213b4649544971308a7a
Jul 17 12:22:37.645492 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.645473    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb"
Jul 17 12:22:37.652564 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:37.652524    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33412f3_025c_4c93_8301_3f3561d98055.slice/crio-4b8eb46b0418010937993e5b0838ba1637cd66d63921c0ccaa389bcabf5958fc WatchSource:0}: Error finding container 4b8eb46b0418010937993e5b0838ba1637cd66d63921c0ccaa389bcabf5958fc: Status 404 returned error can't find the container with id 4b8eb46b0418010937993e5b0838ba1637cd66d63921c0ccaa389bcabf5958fc
Jul 17 12:22:37.657781 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.657767    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m5cxt"
Jul 17 12:22:37.664166 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.664142    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kw5vk"
Jul 17 12:22:37.665096 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:37.665076    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7bee8e_bd18_4de5_938c_a31fd97eb330.slice/crio-b899ab94257656a1cde606c0ebe1fe4470b9ca813c7555af04ec8e2f50dfc58f WatchSource:0}: Error finding container b899ab94257656a1cde606c0ebe1fe4470b9ca813c7555af04ec8e2f50dfc58f: Status 404 returned error can't find the container with id b899ab94257656a1cde606c0ebe1fe4470b9ca813c7555af04ec8e2f50dfc58f
Jul 17 12:22:37.671910 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:37.671885    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0be7fb8_4dda_4b00_9b21_6c105e5bff69.slice/crio-6856f479ef93be8dded50fc31a46e3f72ed2d2817f03c4800b38367f6d2410da WatchSource:0}: Error finding container 6856f479ef93be8dded50fc31a46e3f72ed2d2817f03c4800b38367f6d2410da: Status 404 returned error can't find the container with id 6856f479ef93be8dded50fc31a46e3f72ed2d2817f03c4800b38367f6d2410da
Jul 17 12:22:37.684127 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.684104    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b25ws"
Jul 17 12:22:37.687503 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.687482    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sqgfk"
Jul 17 12:22:37.692972 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:37.692946    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde956422_3a00_4360_87d9_e43b2cb93859.slice/crio-f4f701f12adb13500ef0916ed19b87443acd4149ee0c8b33fcec1d453e6b3f41 WatchSource:0}: Error finding container f4f701f12adb13500ef0916ed19b87443acd4149ee0c8b33fcec1d453e6b3f41: Status 404 returned error can't find the container with id f4f701f12adb13500ef0916ed19b87443acd4149ee0c8b33fcec1d453e6b3f41
Jul 17 12:22:37.696390 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:22:37.696341    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeee8e18_1099_4f80_b580_96a3d6af05a5.slice/crio-25b1f7e7f7f9814d8eaf85a40807437b1caf57fe984e87ee3a47aeb48bac358f WatchSource:0}: Error finding container 25b1f7e7f7f9814d8eaf85a40807437b1caf57fe984e87ee3a47aeb48bac358f: Status 404 returned error can't find the container with id 25b1f7e7f7f9814d8eaf85a40807437b1caf57fe984e87ee3a47aeb48bac358f
Jul 17 12:22:37.710546 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.710526    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 12:22:37.973804 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.973710    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs\") pod \"network-metrics-daemon-ml8cr\" (UID: \"e5f413a7-be80-43f3-950e-be9b5c4bf9b4\") " pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:22:37.973804 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:37.973778    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret\") pod \"global-pull-secret-syncer-vqfdd\" (UID: \"881c9321-df8f-45af-a283-2fe1306b33b5\") " pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:37.974110 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:37.973893    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:37.974110 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:37.973924    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:37.974110 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:37.973970    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs podName:e5f413a7-be80-43f3-950e-be9b5c4bf9b4 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:38.973948693 +0000 UTC m=+3.105333347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs") pod "network-metrics-daemon-ml8cr" (UID: "e5f413a7-be80-43f3-950e-be9b5c4bf9b4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:37.974110 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:37.973994    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret podName:881c9321-df8f-45af-a283-2fe1306b33b5 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:38.97398278 +0000 UTC m=+3.105367427 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret") pod "global-pull-secret-syncer-vqfdd" (UID: "881c9321-df8f-45af-a283-2fe1306b33b5") : object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:38.075228 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:38.074517    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbpv8\" (UniqueName: \"kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8\") pod \"network-check-target-v89v8\" (UID: \"14587634-2d1d-472d-b470-981f1bc0ac8c\") " pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:22:38.075228 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:38.074709    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 12:22:38.075228 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:38.074728    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 12:22:38.075228 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:38.074742    2565 projected.go:196] Error preparing data for projected volume kube-api-access-hbpv8 for pod openshift-network-diagnostics/network-check-target-v89v8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:38.075228 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:38.074807    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8 podName:14587634-2d1d-472d-b470-981f1bc0ac8c nodeName:}" failed. No retries permitted until 2026-07-17 12:22:39.074787849 +0000 UTC m=+3.206172514 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hbpv8" (UniqueName: "kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8") pod "network-check-target-v89v8" (UID: "14587634-2d1d-472d-b470-981f1bc0ac8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:38.334306 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:38.334172    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 12:17:38 +0000 UTC" deadline="2028-04-05 18:43:48.687543056 +0000 UTC"
Jul 17 12:22:38.334306 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:38.334248    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15078h21m10.353301188s"
Jul 17 12:22:38.407895 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:38.407862    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:22:38.408114 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:38.408018    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ml8cr" podUID="e5f413a7-be80-43f3-950e-be9b5c4bf9b4"
Jul 17 12:22:38.426027 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:38.425987    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sqgfk" event={"ID":"eeee8e18-1099-4f80-b580-96a3d6af05a5","Type":"ContainerStarted","Data":"25b1f7e7f7f9814d8eaf85a40807437b1caf57fe984e87ee3a47aeb48bac358f"}
Jul 17 12:22:38.426386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:38.426364    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 12:22:38.435859 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:38.435818    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kw5vk" event={"ID":"d0be7fb8-4dda-4b00-9b21-6c105e5bff69","Type":"ContainerStarted","Data":"6856f479ef93be8dded50fc31a46e3f72ed2d2817f03c4800b38367f6d2410da"}
Jul 17 12:22:38.447038 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:38.447002    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trg5s" event={"ID":"ed8c8549-e57b-4306-b02b-7d1f466f7b4e","Type":"ContainerStarted","Data":"ac424194fef8797a931a21f456ac386b7ee40dac60b2213b4649544971308a7a"}
Jul 17 12:22:38.461825 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:38.461325    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb225" event={"ID":"264b5b0f-2f3e-4acf-af1b-9d731a914efc","Type":"ContainerStarted","Data":"f81a92798e33321e05f8ae9340512975759d09f652b83ddaf1b4bea41bb7309d"}
Jul 17 12:22:38.487422 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:38.487249    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b25ws" event={"ID":"de956422-3a00-4360-87d9-e43b2cb93859","Type":"ContainerStarted","Data":"f4f701f12adb13500ef0916ed19b87443acd4149ee0c8b33fcec1d453e6b3f41"}
Jul 17 12:22:38.508223 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:38.508183    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m5cxt" event={"ID":"dd7bee8e-bd18-4de5-938c-a31fd97eb330","Type":"ContainerStarted","Data":"b899ab94257656a1cde606c0ebe1fe4470b9ca813c7555af04ec8e2f50dfc58f"}
Jul 17 12:22:38.530966 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:38.530876    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb" event={"ID":"f33412f3-025c-4c93-8301-3f3561d98055","Type":"ContainerStarted","Data":"4b8eb46b0418010937993e5b0838ba1637cd66d63921c0ccaa389bcabf5958fc"}
Jul 17 12:22:38.543146 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:38.543110    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b6hrc" event={"ID":"c97f4e9b-753e-4429-b477-03de51ee3fde","Type":"ContainerStarted","Data":"9c9bcf93f995b119af7c0a0729c9d9e9b771921b36022cccf5eb2da0cc890519"}
Jul 17 12:22:38.575244 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:38.575140    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bls5c" event={"ID":"09613d5d-545c-4041-8019-b12539e97032","Type":"ContainerStarted","Data":"1765cf6f2d2ddf3df60696c1d39b065f4994d51f7998a9520fda0590cea4ef34"}
Jul 17 12:22:38.906873 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:38.906838    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 12:22:38.981429 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:38.981387    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs\") pod \"network-metrics-daemon-ml8cr\" (UID: \"e5f413a7-be80-43f3-950e-be9b5c4bf9b4\") " pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:22:38.981610 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:38.981463    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret\") pod \"global-pull-secret-syncer-vqfdd\" (UID: \"881c9321-df8f-45af-a283-2fe1306b33b5\") " pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:38.981677 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:38.981659    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:38.981747 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:38.981732    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret podName:881c9321-df8f-45af-a283-2fe1306b33b5 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:40.981711713 +0000 UTC m=+5.113096358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret") pod "global-pull-secret-syncer-vqfdd" (UID: "881c9321-df8f-45af-a283-2fe1306b33b5") : object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:38.982227 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:38.982204    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:38.982324 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:38.982266    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs podName:e5f413a7-be80-43f3-950e-be9b5c4bf9b4 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:40.982249309 +0000 UTC m=+5.113633956 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs") pod "network-metrics-daemon-ml8cr" (UID: "e5f413a7-be80-43f3-950e-be9b5c4bf9b4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:39.082820 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:39.082774    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbpv8\" (UniqueName: \"kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8\") pod \"network-check-target-v89v8\" (UID: \"14587634-2d1d-472d-b470-981f1bc0ac8c\") " pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:22:39.083028 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:39.082980    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 12:22:39.083028 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:39.083007    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 12:22:39.083028 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:39.083020    2565 projected.go:196] Error preparing data for projected volume kube-api-access-hbpv8 for pod openshift-network-diagnostics/network-check-target-v89v8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:39.083223 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:39.083162    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8 podName:14587634-2d1d-472d-b470-981f1bc0ac8c nodeName:}" failed. No retries permitted until 2026-07-17 12:22:41.083131679 +0000 UTC m=+5.214516323 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hbpv8" (UniqueName: "kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8") pod "network-check-target-v89v8" (UID: "14587634-2d1d-472d-b470-981f1bc0ac8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:39.335609 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:39.335452    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 12:17:38 +0000 UTC" deadline="2028-02-08 05:07:33.652564549 +0000 UTC"
Jul 17 12:22:39.335609 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:39.335520    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13696h44m54.317048991s"
Jul 17 12:22:39.397305 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:39.396575    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:22:39.397305 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:39.396730    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v89v8" podUID="14587634-2d1d-472d-b470-981f1bc0ac8c"
Jul 17 12:22:39.397305 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:39.396826    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:39.397305 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:39.396908    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfdd" podUID="881c9321-df8f-45af-a283-2fe1306b33b5"
Jul 17 12:22:40.402985 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:40.402179    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:22:40.402985 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:40.402566    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ml8cr" podUID="e5f413a7-be80-43f3-950e-be9b5c4bf9b4"
Jul 17 12:22:41.006106 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:41.006065    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs\") pod \"network-metrics-daemon-ml8cr\" (UID: \"e5f413a7-be80-43f3-950e-be9b5c4bf9b4\") " pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:22:41.006288 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:41.006138    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret\") pod \"global-pull-secret-syncer-vqfdd\" (UID: \"881c9321-df8f-45af-a283-2fe1306b33b5\") " pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:41.006288 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:41.006279    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:41.006396 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:41.006346    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret podName:881c9321-df8f-45af-a283-2fe1306b33b5 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:45.006326179 +0000 UTC m=+9.137710824 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret") pod "global-pull-secret-syncer-vqfdd" (UID: "881c9321-df8f-45af-a283-2fe1306b33b5") : object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:41.006769 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:41.006750    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:41.006867 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:41.006810    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs podName:e5f413a7-be80-43f3-950e-be9b5c4bf9b4 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:45.006791979 +0000 UTC m=+9.138176625 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs") pod "network-metrics-daemon-ml8cr" (UID: "e5f413a7-be80-43f3-950e-be9b5c4bf9b4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:41.107416 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:41.107381    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbpv8\" (UniqueName: \"kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8\") pod \"network-check-target-v89v8\" (UID: \"14587634-2d1d-472d-b470-981f1bc0ac8c\") " pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:22:41.107616 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:41.107557    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 12:22:41.107616 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:41.107582    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 12:22:41.107616 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:41.107596    2565 projected.go:196] Error preparing data for projected volume kube-api-access-hbpv8 for pod openshift-network-diagnostics/network-check-target-v89v8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:41.107788 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:41.107665    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8 podName:14587634-2d1d-472d-b470-981f1bc0ac8c nodeName:}" failed. No retries permitted until 2026-07-17 12:22:45.107645789 +0000 UTC m=+9.239030451 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hbpv8" (UniqueName: "kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8") pod "network-check-target-v89v8" (UID: "14587634-2d1d-472d-b470-981f1bc0ac8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:41.396500 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:41.396420    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:22:41.396660 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:41.396553    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v89v8" podUID="14587634-2d1d-472d-b470-981f1bc0ac8c"
Jul 17 12:22:41.397163 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:41.396421    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:41.397163 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:41.397111    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfdd" podUID="881c9321-df8f-45af-a283-2fe1306b33b5"
Jul 17 12:22:42.397165 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:42.397121    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:22:42.397655 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:42.397292    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ml8cr" podUID="e5f413a7-be80-43f3-950e-be9b5c4bf9b4"
Jul 17 12:22:43.396479 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:43.396437    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:22:43.396678 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:43.396564    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v89v8" podUID="14587634-2d1d-472d-b470-981f1bc0ac8c"
Jul 17 12:22:43.397017 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:43.396989    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:43.397136 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:43.397118    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfdd" podUID="881c9321-df8f-45af-a283-2fe1306b33b5"
Jul 17 12:22:44.398306 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:44.398262    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:22:44.398793 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:44.398395    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ml8cr" podUID="e5f413a7-be80-43f3-950e-be9b5c4bf9b4"
Jul 17 12:22:45.043360 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:45.043316    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret\") pod \"global-pull-secret-syncer-vqfdd\" (UID: \"881c9321-df8f-45af-a283-2fe1306b33b5\") " pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:45.043549 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:45.043415    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs\") pod \"network-metrics-daemon-ml8cr\" (UID: \"e5f413a7-be80-43f3-950e-be9b5c4bf9b4\") " pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:22:45.043549 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:45.043514    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:45.043549 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:45.043530    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:45.043736 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:45.043604    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs podName:e5f413a7-be80-43f3-950e-be9b5c4bf9b4 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:53.043582392 +0000 UTC m=+17.174967037 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs") pod "network-metrics-daemon-ml8cr" (UID: "e5f413a7-be80-43f3-950e-be9b5c4bf9b4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:45.043736 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:45.043626    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret podName:881c9321-df8f-45af-a283-2fe1306b33b5 nodeName:}" failed. No retries permitted until 2026-07-17 12:22:53.04361585 +0000 UTC m=+17.175000494 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret") pod "global-pull-secret-syncer-vqfdd" (UID: "881c9321-df8f-45af-a283-2fe1306b33b5") : object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:45.144497 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:45.144418    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbpv8\" (UniqueName: \"kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8\") pod \"network-check-target-v89v8\" (UID: \"14587634-2d1d-472d-b470-981f1bc0ac8c\") " pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:22:45.144693 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:45.144618    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 12:22:45.144693 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:45.144649    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 12:22:45.144693 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:45.144666    2565 projected.go:196] Error preparing data for projected volume kube-api-access-hbpv8 for pod openshift-network-diagnostics/network-check-target-v89v8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:45.144872 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:45.144734    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8 podName:14587634-2d1d-472d-b470-981f1bc0ac8c nodeName:}" failed. No retries permitted until 2026-07-17 12:22:53.144712666 +0000 UTC m=+17.276097320 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hbpv8" (UniqueName: "kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8") pod "network-check-target-v89v8" (UID: "14587634-2d1d-472d-b470-981f1bc0ac8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:45.397129 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:45.397090    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:45.397314 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:45.397214    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfdd" podUID="881c9321-df8f-45af-a283-2fe1306b33b5"
Jul 17 12:22:45.397314 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:45.397282    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:22:45.397440 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:45.397363    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v89v8" podUID="14587634-2d1d-472d-b470-981f1bc0ac8c"
Jul 17 12:22:46.399678 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:46.399639    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:22:46.400102 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:46.399761    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ml8cr" podUID="e5f413a7-be80-43f3-950e-be9b5c4bf9b4"
Jul 17 12:22:47.396926 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:47.396881    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:22:47.397136 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:47.397050    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v89v8" podUID="14587634-2d1d-472d-b470-981f1bc0ac8c"
Jul 17 12:22:47.397207 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:47.397124    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:47.397289 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:47.397254    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfdd" podUID="881c9321-df8f-45af-a283-2fe1306b33b5"
Jul 17 12:22:48.396427 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:48.396393    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:22:48.396922 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:48.396542    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ml8cr" podUID="e5f413a7-be80-43f3-950e-be9b5c4bf9b4"
Jul 17 12:22:49.396339 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:49.396302    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:49.396507 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:49.396438    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfdd" podUID="881c9321-df8f-45af-a283-2fe1306b33b5"
Jul 17 12:22:49.396507 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:49.396493    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:22:49.396826 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:49.396595    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v89v8" podUID="14587634-2d1d-472d-b470-981f1bc0ac8c"
Jul 17 12:22:50.396463 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:50.396435    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:22:50.396604 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:50.396537    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ml8cr" podUID="e5f413a7-be80-43f3-950e-be9b5c4bf9b4"
Jul 17 12:22:51.396514 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:51.396483    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:22:51.396514 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:51.396521    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:51.397059 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:51.396603    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v89v8" podUID="14587634-2d1d-472d-b470-981f1bc0ac8c"
Jul 17 12:22:51.397059 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:51.396724    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfdd" podUID="881c9321-df8f-45af-a283-2fe1306b33b5"
Jul 17 12:22:52.397111 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:52.397074    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:22:52.397566 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:52.397226    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ml8cr" podUID="e5f413a7-be80-43f3-950e-be9b5c4bf9b4"
Jul 17 12:22:53.112387 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:53.112349    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret\") pod \"global-pull-secret-syncer-vqfdd\" (UID: \"881c9321-df8f-45af-a283-2fe1306b33b5\") " pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:53.112607 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:53.112420    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs\") pod \"network-metrics-daemon-ml8cr\" (UID: \"e5f413a7-be80-43f3-950e-be9b5c4bf9b4\") " pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:22:53.112607 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:53.112506    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:53.112607 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:53.112506    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:53.112607 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:53.112571    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs podName:e5f413a7-be80-43f3-950e-be9b5c4bf9b4 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:09.112550487 +0000 UTC m=+33.243935138 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs") pod "network-metrics-daemon-ml8cr" (UID: "e5f413a7-be80-43f3-950e-be9b5c4bf9b4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:22:53.112607 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:53.112587    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret podName:881c9321-df8f-45af-a283-2fe1306b33b5 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:09.112578943 +0000 UTC m=+33.243963591 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret") pod "global-pull-secret-syncer-vqfdd" (UID: "881c9321-df8f-45af-a283-2fe1306b33b5") : object "kube-system"/"original-pull-secret" not registered
Jul 17 12:22:53.213095 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:53.213051    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbpv8\" (UniqueName: \"kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8\") pod \"network-check-target-v89v8\" (UID: \"14587634-2d1d-472d-b470-981f1bc0ac8c\") " pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:22:53.213268 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:53.213238    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 12:22:53.213337 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:53.213268    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 12:22:53.213337 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:53.213280    2565 projected.go:196] Error preparing data for projected volume kube-api-access-hbpv8 for pod openshift-network-diagnostics/network-check-target-v89v8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:53.213448 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:53.213340    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8 podName:14587634-2d1d-472d-b470-981f1bc0ac8c nodeName:}" failed. No retries permitted until 2026-07-17 12:23:09.213314987 +0000 UTC m=+33.344699626 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hbpv8" (UniqueName: "kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8") pod "network-check-target-v89v8" (UID: "14587634-2d1d-472d-b470-981f1bc0ac8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:22:53.396383 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:53.396293    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:22:53.396538 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:53.396293    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:53.396538 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:53.396421    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v89v8" podUID="14587634-2d1d-472d-b470-981f1bc0ac8c"
Jul 17 12:22:53.396538 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:53.396513    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfdd" podUID="881c9321-df8f-45af-a283-2fe1306b33b5"
Jul 17 12:22:54.396265 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:54.396223    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:22:54.396662 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:54.396385    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ml8cr" podUID="e5f413a7-be80-43f3-950e-be9b5c4bf9b4"
Jul 17 12:22:55.396509 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:55.396473    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:22:55.396934 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:55.396487    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:55.396934 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:55.396602    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v89v8" podUID="14587634-2d1d-472d-b470-981f1bc0ac8c"
Jul 17 12:22:55.396934 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:55.396674    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfdd" podUID="881c9321-df8f-45af-a283-2fe1306b33b5"
Jul 17 12:22:56.397585 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:56.397549    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:22:56.398021 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:56.397679    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ml8cr" podUID="e5f413a7-be80-43f3-950e-be9b5c4bf9b4"
Jul 17 12:22:57.396668 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:57.396637    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:57.396864 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:57.396763    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfdd" podUID="881c9321-df8f-45af-a283-2fe1306b33b5"
Jul 17 12:22:57.396864 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:57.396826    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:22:57.397005 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:57.396918    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v89v8" podUID="14587634-2d1d-472d-b470-981f1bc0ac8c"
Jul 17 12:22:58.396282 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:58.396253    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:22:58.396751 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:58.396403    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ml8cr" podUID="e5f413a7-be80-43f3-950e-be9b5c4bf9b4"
Jul 17 12:22:59.397069 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:59.397043    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:22:59.397390 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:59.397166    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfdd" podUID="881c9321-df8f-45af-a283-2fe1306b33b5"
Jul 17 12:22:59.397450 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:59.397429    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:22:59.397566 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:22:59.397543    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v89v8" podUID="14587634-2d1d-472d-b470-981f1bc0ac8c"
Jul 17 12:22:59.631824 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:59.631620    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sqgfk" event={"ID":"eeee8e18-1099-4f80-b580-96a3d6af05a5","Type":"ContainerStarted","Data":"7450116859b0ded48636623d46470a8bdabb4fa0f60cd56130d8b161929710b0"}
Jul 17 12:22:59.636405 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:59.636383    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trg5s" event={"ID":"ed8c8549-e57b-4306-b02b-7d1f466f7b4e","Type":"ContainerStarted","Data":"74f65ab8a241b5e3b5c1bee7021c7ff261d4bde2c704fff174c41f5afe835900"}
Jul 17 12:22:59.636523 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:59.636449    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trg5s" event={"ID":"ed8c8549-e57b-4306-b02b-7d1f466f7b4e","Type":"ContainerStarted","Data":"16b4f9924cccd4d6051456ceb791ed4b2ee5962d30cd7bea65928fde9c6b6fd7"}
Jul 17 12:22:59.637637 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:59.637616    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-119.ec2.internal" event={"ID":"9e932a7529a4a2a40c2cedf7c627d21a","Type":"ContainerStarted","Data":"2f15f9aeedd21b9e5e86a68368c9caec27be2f4040953323db80604ca9aa230d"}
Jul 17 12:22:59.638953 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:59.638929    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m5cxt" event={"ID":"dd7bee8e-bd18-4de5-938c-a31fd97eb330","Type":"ContainerStarted","Data":"ba709e2203a5864a0883c91938eff5b97e5a9171898210af886b6b99f52048ad"}
Jul 17 12:22:59.658630 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:59.658594    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-sqgfk" podStartSLOduration=1.09334241 podStartE2EDuration="22.658566633s" totalImagesPullingTime="21.565224223s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:59.65363258 +0000 UTC m=+23.785017240" watchObservedRunningTime="2026-07-17 12:22:59.658566633 +0000 UTC m=+23.789951288"
Jul 17 12:22:59.670200 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:59.670153    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-m5cxt" podStartSLOduration=1.082241837 podStartE2EDuration="22.670136314s" totalImagesPullingTime="21.587894477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:59.669333084 +0000 UTC m=+23.800717734" watchObservedRunningTime="2026-07-17 12:22:59.670136314 +0000 UTC m=+23.801520975"
Jul 17 12:22:59.682434 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:22:59.682401    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-119.ec2.internal" podStartSLOduration=21.68238954 podStartE2EDuration="21.68238954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:22:59.682059933 +0000 UTC m=+23.813444594" watchObservedRunningTime="2026-07-17 12:22:59.68238954 +0000 UTC m=+23.813774201"
Jul 17 12:23:00.396793 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:00.396566    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:23:00.396935 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:00.396821    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ml8cr" podUID="e5f413a7-be80-43f3-950e-be9b5c4bf9b4"
Jul 17 12:23:00.643245 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:00.643217    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 12:23:00.644119 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:00.643548    2565 generic.go:356] "Generic (PLEG): container finished" podID="ed8c8549-e57b-4306-b02b-7d1f466f7b4e" containerID="74f65ab8a241b5e3b5c1bee7021c7ff261d4bde2c704fff174c41f5afe835900" exitCode=1
Jul 17 12:23:00.644119 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:00.643636    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trg5s" event={"ID":"ed8c8549-e57b-4306-b02b-7d1f466f7b4e","Type":"ContainerDied","Data":"74f65ab8a241b5e3b5c1bee7021c7ff261d4bde2c704fff174c41f5afe835900"}
Jul 17 12:23:00.644119 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:00.643719    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trg5s" event={"ID":"ed8c8549-e57b-4306-b02b-7d1f466f7b4e","Type":"ContainerStarted","Data":"319c2ba92382e031ad84ba490211c668b181ddba1bf2b95507403132311a2227"}
Jul 17 12:23:00.644119 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:00.643763    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trg5s" event={"ID":"ed8c8549-e57b-4306-b02b-7d1f466f7b4e","Type":"ContainerStarted","Data":"a03c7d8e297fc40de4e1a3a5b3e018197f517f938fbc43f8f54b920c6236c0ab"}
Jul 17 12:23:00.644119 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:00.643808    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trg5s" event={"ID":"ed8c8549-e57b-4306-b02b-7d1f466f7b4e","Type":"ContainerStarted","Data":"ebc24f16a366d95f5f1c38aefefb7c83a81f7f9765299ef16f3529a6ed5dd2cc"}
Jul 17 12:23:00.644119 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:00.643851    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trg5s" event={"ID":"ed8c8549-e57b-4306-b02b-7d1f466f7b4e","Type":"ContainerStarted","Data":"0897dc7b4a281776872d3d8b02e33f6265303467b23ba52a3ea023514a35f292"}
Jul 17 12:23:00.645088 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:00.645068    2565 generic.go:356] "Generic (PLEG): container finished" podID="264b5b0f-2f3e-4acf-af1b-9d731a914efc" containerID="434e99df4171514dc501101cd0ef5bd0f7d33bb93377a75093447a73615a338b" exitCode=0
Jul 17 12:23:00.645215 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:00.645134    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb225" event={"ID":"264b5b0f-2f3e-4acf-af1b-9d731a914efc","Type":"ContainerDied","Data":"434e99df4171514dc501101cd0ef5bd0f7d33bb93377a75093447a73615a338b"}
Jul 17 12:23:00.646652 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:00.646629    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b25ws" event={"ID":"de956422-3a00-4360-87d9-e43b2cb93859","Type":"ContainerStarted","Data":"b4dd3104a02c1f24133fb85852db38048e78ee3d119f81088decc80288b453bd"}
Jul 17 12:23:00.648056 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:00.648002    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb" event={"ID":"f33412f3-025c-4c93-8301-3f3561d98055","Type":"ContainerStarted","Data":"655c2822a60548be04ce72ee0286509cb99904c1a8338898e822f0521d52f02e"}
Jul 17 12:23:00.649491 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:00.649467    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b6hrc" event={"ID":"c97f4e9b-753e-4429-b477-03de51ee3fde","Type":"ContainerStarted","Data":"a8dd5aaf43afb565abb2cbf65bf1ac8002fc4d8799941692f48bb904d5bba9f4"}
Jul 17 12:23:00.650860 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:00.650839    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bls5c" event={"ID":"09613d5d-545c-4041-8019-b12539e97032","Type":"ContainerStarted","Data":"5ffafff593ffa3fc2617c8b329e13b9b305b20731a501d8dc5626fda22ab4eb9"}
Jul 17 12:23:00.652358 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:00.652337    2565 generic.go:356] "Generic (PLEG): container finished" podID="40802730aa52aba45795c84facc5eccc" containerID="54a03928e916a73b167ba6aa4798178d84a56951f23cd5291066b864df2fb817" exitCode=0
Jul 17 12:23:00.652439 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:00.652402    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-119.ec2.internal" event={"ID":"40802730aa52aba45795c84facc5eccc","Type":"ContainerDied","Data":"54a03928e916a73b167ba6aa4798178d84a56951f23cd5291066b864df2fb817"}
Jul 17 12:23:00.653832 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:00.653812    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kw5vk" event={"ID":"d0be7fb8-4dda-4b00-9b21-6c105e5bff69","Type":"ContainerStarted","Data":"243de06ba3974ac0b7a1496fc9bed4b0d25851eb4563d44abf455add55eb5fc8"}
Jul 17 12:23:00.683103 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:00.683037    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kw5vk" podStartSLOduration=2.100899961 podStartE2EDuration="23.683019938s" totalImagesPullingTime="21.582119977s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:23:00.682561705 +0000 UTC m=+24.813946367" watchObservedRunningTime="2026-07-17 12:23:00.683019938 +0000 UTC m=+24.814404600"
Jul 17 12:23:00.697116 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:00.697080    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-b25ws" podStartSLOduration=2.1584562959999998 podStartE2EDuration="23.697067884s" totalImagesPullingTime="21.538611588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:23:00.696706234 +0000 UTC m=+24.828090909" watchObservedRunningTime="2026-07-17 12:23:00.697067884 +0000 UTC m=+24.828452545"
Jul 17 12:23:00.724618 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:00.724583    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bls5c" podStartSLOduration=2.07694498 podStartE2EDuration="23.724564337s" totalImagesPullingTime="21.647619357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:23:00.724222645 +0000 UTC m=+24.855607305" watchObservedRunningTime="2026-07-17 12:23:00.724564337 +0000 UTC m=+24.855948999"
Jul 17 12:23:00.742094 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:00.742048    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b6hrc" podStartSLOduration=2.132276234 podStartE2EDuration="23.742029017s" totalImagesPullingTime="21.609752783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:23:00.741631841 +0000 UTC m=+24.873016502" watchObservedRunningTime="2026-07-17 12:23:00.742029017 +0000 UTC m=+24.873413674"
Jul 17 12:23:01.298163 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:01.298133    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 17 12:23:01.337799 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:01.337686    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T12:23:01.298154559Z","UUID":"51077571-3994-4bd3-aff6-68c1e60b5907","Handler":null,"Name":"","Endpoint":""}
Jul 17 12:23:01.339515 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:01.339487    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 17 12:23:01.339626 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:01.339534    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 17 12:23:01.396577 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:01.396546    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:23:01.396774 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:01.396661    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfdd" podUID="881c9321-df8f-45af-a283-2fe1306b33b5"
Jul 17 12:23:01.396774 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:01.396725    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:23:01.396873 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:01.396835    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v89v8" podUID="14587634-2d1d-472d-b470-981f1bc0ac8c"
Jul 17 12:23:01.658176 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:01.658144    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb" event={"ID":"f33412f3-025c-4c93-8301-3f3561d98055","Type":"ContainerStarted","Data":"06f5e0457ae18f8c399366691550f3fcdca43266775c1106dd7632f62f9b638e"}
Jul 17 12:23:01.659898 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:01.659866    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-119.ec2.internal" event={"ID":"40802730aa52aba45795c84facc5eccc","Type":"ContainerStarted","Data":"0b2fe10bd74cdf325d48bb0de1abfa1902eaf25684cf62aa8ec528dcaf9dd42f"}
Jul 17 12:23:01.674946 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:01.674899    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-119.ec2.internal" podStartSLOduration=23.674880599 podStartE2EDuration="23.674880599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:23:01.674544015 +0000 UTC m=+25.805928675" watchObservedRunningTime="2026-07-17 12:23:01.674880599 +0000 UTC m=+25.806265276"
Jul 17 12:23:02.396583 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:02.396524    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:23:02.396732 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:02.396644    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ml8cr" podUID="e5f413a7-be80-43f3-950e-be9b5c4bf9b4"
Jul 17 12:23:02.665174 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:02.665113    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 12:23:02.665549 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:02.665526    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trg5s" event={"ID":"ed8c8549-e57b-4306-b02b-7d1f466f7b4e","Type":"ContainerStarted","Data":"74843d392da37dafc77853fddb0ecdbed8004852246ad78dc71f5e707ade744f"}
Jul 17 12:23:02.667577 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:02.667551    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb" event={"ID":"f33412f3-025c-4c93-8301-3f3561d98055","Type":"ContainerStarted","Data":"3775921d059bc64011e37562cc3bd611b5e80f7175b478eb7f3204ee29f2f672"}
Jul 17 12:23:02.685120 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:02.684915    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nmxkb" podStartSLOduration=1.9367316030000001 podStartE2EDuration="25.68490264s" totalImagesPullingTime="23.748171037s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:23:02.684824811 +0000 UTC m=+26.816209473" watchObservedRunningTime="2026-07-17 12:23:02.68490264 +0000 UTC m=+26.816287322"
Jul 17 12:23:03.396887 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:03.396851    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:23:03.397101 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:03.396961    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfdd" podUID="881c9321-df8f-45af-a283-2fe1306b33b5"
Jul 17 12:23:03.397101 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:03.397026    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:23:03.397225 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:03.397124    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v89v8" podUID="14587634-2d1d-472d-b470-981f1bc0ac8c"
Jul 17 12:23:04.399353 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:04.399324    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:23:04.399790 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:04.399424    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ml8cr" podUID="e5f413a7-be80-43f3-950e-be9b5c4bf9b4"
Jul 17 12:23:05.064483 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:05.064454    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b6hrc"
Jul 17 12:23:05.396585 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:05.396559    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:23:05.396671 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:05.396658    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfdd" podUID="881c9321-df8f-45af-a283-2fe1306b33b5"
Jul 17 12:23:05.396737 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:05.396716    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:23:05.396838 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:05.396821    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v89v8" podUID="14587634-2d1d-472d-b470-981f1bc0ac8c"
Jul 17 12:23:05.600236 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:05.600007    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b6hrc"
Jul 17 12:23:05.600725 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:05.600494    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b6hrc"
Jul 17 12:23:05.677538 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:05.677510    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 12:23:05.677918 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:05.677892    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trg5s" event={"ID":"ed8c8549-e57b-4306-b02b-7d1f466f7b4e","Type":"ContainerStarted","Data":"7d226348ab596bf213eaf0521ea524061f21031039662a6870cae11204a3ef55"}
Jul 17 12:23:05.678384 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:05.678361    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:23:05.678533 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:05.678511    2565 scope.go:117] "RemoveContainer" containerID="74f65ab8a241b5e3b5c1bee7021c7ff261d4bde2c704fff174c41f5afe835900"
Jul 17 12:23:05.680124 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:05.680100    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb225" event={"ID":"264b5b0f-2f3e-4acf-af1b-9d731a914efc","Type":"ContainerStarted","Data":"7a6fd9b43ef8878b7f06359965707bdb268706a43e3d5c98918304b9ab02daca"}
Jul 17 12:23:05.680732 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:05.680712    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b6hrc"
Jul 17 12:23:05.695108 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:05.695082    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:23:06.397695 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:06.397659    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:23:06.397860 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:06.397794    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ml8cr" podUID="e5f413a7-be80-43f3-950e-be9b5c4bf9b4"
Jul 17 12:23:06.685966 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:06.685939    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 12:23:06.686445 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:06.686381    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trg5s" event={"ID":"ed8c8549-e57b-4306-b02b-7d1f466f7b4e","Type":"ContainerStarted","Data":"c117059c8da642e929794d2be0097a9f5f6f8bc1a179ad8615cc7f806ee6e47e"}
Jul 17 12:23:06.686697 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:06.686669    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:23:06.686780 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:06.686729    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:23:06.688066 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:06.688045    2565 generic.go:356] "Generic (PLEG): container finished" podID="264b5b0f-2f3e-4acf-af1b-9d731a914efc" containerID="7a6fd9b43ef8878b7f06359965707bdb268706a43e3d5c98918304b9ab02daca" exitCode=0
Jul 17 12:23:06.688164 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:06.688126    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb225" event={"ID":"264b5b0f-2f3e-4acf-af1b-9d731a914efc","Type":"ContainerDied","Data":"7a6fd9b43ef8878b7f06359965707bdb268706a43e3d5c98918304b9ab02daca"}
Jul 17 12:23:06.703496 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:06.703463    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:23:06.713692 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:06.713649    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-trg5s" podStartSLOduration=8.047986315 podStartE2EDuration="29.713636294s" totalImagesPullingTime="21.665649979s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:23:06.712002722 +0000 UTC m=+30.843387400" watchObservedRunningTime="2026-07-17 12:23:06.713636294 +0000 UTC m=+30.845020954"
Jul 17 12:23:07.285918 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:07.285871    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vqfdd"]
Jul 17 12:23:07.286256 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:07.286082    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:23:07.286256 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:07.286189    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfdd" podUID="881c9321-df8f-45af-a283-2fe1306b33b5"
Jul 17 12:23:07.286644 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:07.286626    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v89v8"]
Jul 17 12:23:07.286736 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:07.286723    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:23:07.286819 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:07.286803    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v89v8" podUID="14587634-2d1d-472d-b470-981f1bc0ac8c"
Jul 17 12:23:07.297200 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:07.297175    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ml8cr"]
Jul 17 12:23:07.297341 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:07.297294    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:23:07.297446 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:07.297411    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ml8cr" podUID="e5f413a7-be80-43f3-950e-be9b5c4bf9b4"
Jul 17 12:23:08.397340 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:08.397090    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:23:08.397709 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:08.397448    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v89v8" podUID="14587634-2d1d-472d-b470-981f1bc0ac8c"
Jul 17 12:23:08.698435 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:08.698320    2565 generic.go:356] "Generic (PLEG): container finished" podID="264b5b0f-2f3e-4acf-af1b-9d731a914efc" containerID="1c9d2e82f5e8185ed6833f487daddf7642de57c088313b0f44031f5b24f628cd" exitCode=0
Jul 17 12:23:08.698592 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:08.698441    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb225" event={"ID":"264b5b0f-2f3e-4acf-af1b-9d731a914efc","Type":"ContainerDied","Data":"1c9d2e82f5e8185ed6833f487daddf7642de57c088313b0f44031f5b24f628cd"}
Jul 17 12:23:09.142699 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:09.142659    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret\") pod \"global-pull-secret-syncer-vqfdd\" (UID: \"881c9321-df8f-45af-a283-2fe1306b33b5\") " pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:23:09.142883 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:09.142718    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs\") pod \"network-metrics-daemon-ml8cr\" (UID: \"e5f413a7-be80-43f3-950e-be9b5c4bf9b4\") " pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:23:09.142883 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:09.142814    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:23:09.142883 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:09.142816    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 12:23:09.142883 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:09.142871    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs podName:e5f413a7-be80-43f3-950e-be9b5c4bf9b4 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:41.142852756 +0000 UTC m=+65.274237397 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs") pod "network-metrics-daemon-ml8cr" (UID: "e5f413a7-be80-43f3-950e-be9b5c4bf9b4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 12:23:09.142883 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:09.142887    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret podName:881c9321-df8f-45af-a283-2fe1306b33b5 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:41.142879846 +0000 UTC m=+65.274264484 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret") pod "global-pull-secret-syncer-vqfdd" (UID: "881c9321-df8f-45af-a283-2fe1306b33b5") : object "kube-system"/"original-pull-secret" not registered
Jul 17 12:23:09.244062 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:09.243974    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbpv8\" (UniqueName: \"kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8\") pod \"network-check-target-v89v8\" (UID: \"14587634-2d1d-472d-b470-981f1bc0ac8c\") " pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:23:09.244200 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:09.244125    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 12:23:09.244200 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:09.244145    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 12:23:09.244200 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:09.244155    2565 projected.go:196] Error preparing data for projected volume kube-api-access-hbpv8 for pod openshift-network-diagnostics/network-check-target-v89v8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:23:09.244315 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:09.244204    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8 podName:14587634-2d1d-472d-b470-981f1bc0ac8c nodeName:}" failed. No retries permitted until 2026-07-17 12:23:41.244190724 +0000 UTC m=+65.375575366 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hbpv8" (UniqueName: "kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8") pod "network-check-target-v89v8" (UID: "14587634-2d1d-472d-b470-981f1bc0ac8c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 12:23:09.397250 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:09.397219    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:23:09.397396 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:09.397223    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:23:09.397396 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:09.397330    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfdd" podUID="881c9321-df8f-45af-a283-2fe1306b33b5"
Jul 17 12:23:09.397921 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:09.397443    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ml8cr" podUID="e5f413a7-be80-43f3-950e-be9b5c4bf9b4"
Jul 17 12:23:10.396569 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:10.396539    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:23:10.396752 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:10.396648    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v89v8" podUID="14587634-2d1d-472d-b470-981f1bc0ac8c"
Jul 17 12:23:10.704254 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:10.704217    2565 generic.go:356] "Generic (PLEG): container finished" podID="264b5b0f-2f3e-4acf-af1b-9d731a914efc" containerID="dd75e7ceba4e4fc7038502ce05de87cd94186ec9d50f773abd30e2e52a73a44d" exitCode=0
Jul 17 12:23:10.704689 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:10.704253    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb225" event={"ID":"264b5b0f-2f3e-4acf-af1b-9d731a914efc","Type":"ContainerDied","Data":"dd75e7ceba4e4fc7038502ce05de87cd94186ec9d50f773abd30e2e52a73a44d"}
Jul 17 12:23:11.396394 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:11.396365    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:23:11.396622 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:11.396365    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:23:11.396622 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:11.396462    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vqfdd" podUID="881c9321-df8f-45af-a283-2fe1306b33b5"
Jul 17 12:23:11.396622 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:11.396557    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ml8cr" podUID="e5f413a7-be80-43f3-950e-be9b5c4bf9b4"
Jul 17 12:23:12.217086 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.217055    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-119.ec2.internal" event="NodeReady"
Jul 17 12:23:12.217637 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.217203    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 17 12:23:12.260640 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.260616    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gbnmx"]
Jul 17 12:23:12.262734 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.262714    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:23:12.263047 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.263027    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n7hp4"]
Jul 17 12:23:12.264877 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.264864    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n7hp4"
Jul 17 12:23:12.265363 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.265342    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 17 12:23:12.265914 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.265898    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gk42k\""
Jul 17 12:23:12.266021 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.265900    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 17 12:23:12.267546 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.267525    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 17 12:23:12.267750 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.267737    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 17 12:23:12.268094 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.268078    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-btjhs\""
Jul 17 12:23:12.268153 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.268118    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 17 12:23:12.277832 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.277814    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gbnmx"]
Jul 17 12:23:12.278356 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.278340    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n7hp4"]
Jul 17 12:23:12.368756 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.368725    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02906c20-b77c-4c91-b4e4-fcd72c647670-cert\") pod \"ingress-canary-n7hp4\" (UID: \"02906c20-b77c-4c91-b4e4-fcd72c647670\") " pod="openshift-ingress-canary/ingress-canary-n7hp4"
Jul 17 12:23:12.368909 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.368766    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793d61b2-520e-416a-b856-545c0a0f45b5-tmp-dir\") pod \"dns-default-gbnmx\" (UID: \"793d61b2-520e-416a-b856-545c0a0f45b5\") " pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:23:12.368909 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.368785    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlfjd\" (UniqueName: \"kubernetes.io/projected/02906c20-b77c-4c91-b4e4-fcd72c647670-kube-api-access-tlfjd\") pod \"ingress-canary-n7hp4\" (UID: \"02906c20-b77c-4c91-b4e4-fcd72c647670\") " pod="openshift-ingress-canary/ingress-canary-n7hp4"
Jul 17 12:23:12.368909 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.368804    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/793d61b2-520e-416a-b856-545c0a0f45b5-config-volume\") pod \"dns-default-gbnmx\" (UID: \"793d61b2-520e-416a-b856-545c0a0f45b5\") " pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:23:12.368909 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.368845    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls\") pod \"dns-default-gbnmx\" (UID: \"793d61b2-520e-416a-b856-545c0a0f45b5\") " pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:23:12.368909 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.368882    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wxxp\" (UniqueName: \"kubernetes.io/projected/793d61b2-520e-416a-b856-545c0a0f45b5-kube-api-access-4wxxp\") pod \"dns-default-gbnmx\" (UID: \"793d61b2-520e-416a-b856-545c0a0f45b5\") " pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:23:12.397204 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.397177    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:23:12.400222 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.400203    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2chv5\""
Jul 17 12:23:12.400339 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.400209    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 12:23:12.400339 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.400250    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 12:23:12.469451 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.469425    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wxxp\" (UniqueName: \"kubernetes.io/projected/793d61b2-520e-416a-b856-545c0a0f45b5-kube-api-access-4wxxp\") pod \"dns-default-gbnmx\" (UID: \"793d61b2-520e-416a-b856-545c0a0f45b5\") " pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:23:12.469595 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.469464    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02906c20-b77c-4c91-b4e4-fcd72c647670-cert\") pod \"ingress-canary-n7hp4\" (UID: \"02906c20-b77c-4c91-b4e4-fcd72c647670\") " pod="openshift-ingress-canary/ingress-canary-n7hp4"
Jul 17 12:23:12.469595 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.469500    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793d61b2-520e-416a-b856-545c0a0f45b5-tmp-dir\") pod \"dns-default-gbnmx\" (UID: \"793d61b2-520e-416a-b856-545c0a0f45b5\") " pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:23:12.469595 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.469523    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlfjd\" (UniqueName: \"kubernetes.io/projected/02906c20-b77c-4c91-b4e4-fcd72c647670-kube-api-access-tlfjd\") pod \"ingress-canary-n7hp4\" (UID: \"02906c20-b77c-4c91-b4e4-fcd72c647670\") " pod="openshift-ingress-canary/ingress-canary-n7hp4"
Jul 17 12:23:12.469595 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.469545    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/793d61b2-520e-416a-b856-545c0a0f45b5-config-volume\") pod \"dns-default-gbnmx\" (UID: \"793d61b2-520e-416a-b856-545c0a0f45b5\") " pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:23:12.469595 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.469588    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls\") pod \"dns-default-gbnmx\" (UID: \"793d61b2-520e-416a-b856-545c0a0f45b5\") " pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:23:12.469841 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:12.469701    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:23:12.469841 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:12.469768    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls podName:793d61b2-520e-416a-b856-545c0a0f45b5 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:12.969749515 +0000 UTC m=+37.101134173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls") pod "dns-default-gbnmx" (UID: "793d61b2-520e-416a-b856-545c0a0f45b5") : secret "dns-default-metrics-tls" not found
Jul 17 12:23:12.469942 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.469879    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/793d61b2-520e-416a-b856-545c0a0f45b5-tmp-dir\") pod \"dns-default-gbnmx\" (UID: \"793d61b2-520e-416a-b856-545c0a0f45b5\") " pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:23:12.470198 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.470180    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/793d61b2-520e-416a-b856-545c0a0f45b5-config-volume\") pod \"dns-default-gbnmx\" (UID: \"793d61b2-520e-416a-b856-545c0a0f45b5\") " pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:23:12.474412 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.474394    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02906c20-b77c-4c91-b4e4-fcd72c647670-cert\") pod \"ingress-canary-n7hp4\" (UID: \"02906c20-b77c-4c91-b4e4-fcd72c647670\") " pod="openshift-ingress-canary/ingress-canary-n7hp4"
Jul 17 12:23:12.478394 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.478371    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlfjd\" (UniqueName: \"kubernetes.io/projected/02906c20-b77c-4c91-b4e4-fcd72c647670-kube-api-access-tlfjd\") pod \"ingress-canary-n7hp4\" (UID: \"02906c20-b77c-4c91-b4e4-fcd72c647670\") " pod="openshift-ingress-canary/ingress-canary-n7hp4"
Jul 17 12:23:12.479015 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.478977    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wxxp\" (UniqueName: \"kubernetes.io/projected/793d61b2-520e-416a-b856-545c0a0f45b5-kube-api-access-4wxxp\") pod \"dns-default-gbnmx\" (UID: \"793d61b2-520e-416a-b856-545c0a0f45b5\") " pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:23:12.578520 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.578483    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n7hp4"
Jul 17 12:23:12.749736 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.749551    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n7hp4"]
Jul 17 12:23:12.754587 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:23:12.754553    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02906c20_b77c_4c91_b4e4_fcd72c647670.slice/crio-679b42b6d3e2d915b383c325caf0d7078dce9983a72fb369f1070cb398d2d2b2 WatchSource:0}: Error finding container 679b42b6d3e2d915b383c325caf0d7078dce9983a72fb369f1070cb398d2d2b2: Status 404 returned error can't find the container with id 679b42b6d3e2d915b383c325caf0d7078dce9983a72fb369f1070cb398d2d2b2
Jul 17 12:23:12.973251 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:12.973165    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls\") pod \"dns-default-gbnmx\" (UID: \"793d61b2-520e-416a-b856-545c0a0f45b5\") " pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:23:12.973399 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:12.973315    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:23:12.973399 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:12.973392    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls podName:793d61b2-520e-416a-b856-545c0a0f45b5 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:13.973375213 +0000 UTC m=+38.104759852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls") pod "dns-default-gbnmx" (UID: "793d61b2-520e-416a-b856-545c0a0f45b5") : secret "dns-default-metrics-tls" not found
Jul 17 12:23:13.397074 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:13.397034    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:23:13.397541 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:13.397044    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:23:13.400503 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:13.400479    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 12:23:13.400664 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:13.400651    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 12:23:13.401560 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:13.401543    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vpz7h\""
Jul 17 12:23:13.713406 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:13.713305    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n7hp4" event={"ID":"02906c20-b77c-4c91-b4e4-fcd72c647670","Type":"ContainerStarted","Data":"679b42b6d3e2d915b383c325caf0d7078dce9983a72fb369f1070cb398d2d2b2"}
Jul 17 12:23:13.980784 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:13.980703    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls\") pod \"dns-default-gbnmx\" (UID: \"793d61b2-520e-416a-b856-545c0a0f45b5\") " pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:23:13.980935 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:13.980859    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:23:13.981016 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:13.980959    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls podName:793d61b2-520e-416a-b856-545c0a0f45b5 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:15.980939213 +0000 UTC m=+40.112323855 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls") pod "dns-default-gbnmx" (UID: "793d61b2-520e-416a-b856-545c0a0f45b5") : secret "dns-default-metrics-tls" not found
Jul 17 12:23:15.996947 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:15.996906    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls\") pod \"dns-default-gbnmx\" (UID: \"793d61b2-520e-416a-b856-545c0a0f45b5\") " pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:23:15.997435 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:15.997064    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:23:15.997435 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:15.997126    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls podName:793d61b2-520e-416a-b856-545c0a0f45b5 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:19.997108214 +0000 UTC m=+44.128492860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls") pod "dns-default-gbnmx" (UID: "793d61b2-520e-416a-b856-545c0a0f45b5") : secret "dns-default-metrics-tls" not found
Jul 17 12:23:17.726958 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:17.726928    2565 generic.go:356] "Generic (PLEG): container finished" podID="264b5b0f-2f3e-4acf-af1b-9d731a914efc" containerID="abaf05d8013464b4f330ee6b923e7419e3cffcf76de8777ca1cce525ebc28469" exitCode=0
Jul 17 12:23:17.727381 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:17.727015    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb225" event={"ID":"264b5b0f-2f3e-4acf-af1b-9d731a914efc","Type":"ContainerDied","Data":"abaf05d8013464b4f330ee6b923e7419e3cffcf76de8777ca1cce525ebc28469"}
Jul 17 12:23:17.728470 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:17.728447    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n7hp4" event={"ID":"02906c20-b77c-4c91-b4e4-fcd72c647670","Type":"ContainerStarted","Data":"40f60f6cabb5467cf3df649cdf4d0c1ec3de02c25ff3f025bb079329e6110152"}
Jul 17 12:23:17.764033 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:17.763678    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n7hp4" podStartSLOduration=1.9086291119999999 podStartE2EDuration="5.763659857s" totalImagesPullingTime="3.855030745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:23:17.762308209 +0000 UTC m=+41.893692871" watchObservedRunningTime="2026-07-17 12:23:17.763659857 +0000 UTC m=+41.895044519"
Jul 17 12:23:18.733161 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:18.733124    2565 generic.go:356] "Generic (PLEG): container finished" podID="264b5b0f-2f3e-4acf-af1b-9d731a914efc" containerID="4282048c366418586eac7dd797c2415c5ebe33223d851c58b7208cc102f4052d" exitCode=0
Jul 17 12:23:18.733547 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:18.733204    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb225" event={"ID":"264b5b0f-2f3e-4acf-af1b-9d731a914efc","Type":"ContainerDied","Data":"4282048c366418586eac7dd797c2415c5ebe33223d851c58b7208cc102f4052d"}
Jul 17 12:23:19.738520 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:19.738481    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nb225" event={"ID":"264b5b0f-2f3e-4acf-af1b-9d731a914efc","Type":"ContainerStarted","Data":"4c6dbef4727dc2a5be8374a6a9d67687ea1c7fd8b97bf6cc47e9a897dcf96e36"}
Jul 17 12:23:19.761710 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:19.761660    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nb225" podStartSLOduration=8.177930696 podStartE2EDuration="42.761644898s" totalImagesPullingTime="34.583714202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:23:19.760186201 +0000 UTC m=+43.891570863" watchObservedRunningTime="2026-07-17 12:23:19.761644898 +0000 UTC m=+43.893029559"
Jul 17 12:23:20.022634 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:20.022541    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls\") pod \"dns-default-gbnmx\" (UID: \"793d61b2-520e-416a-b856-545c0a0f45b5\") " pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:23:20.022777 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:20.022663    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:23:20.022777 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:20.022718    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls podName:793d61b2-520e-416a-b856-545c0a0f45b5 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:28.022701916 +0000 UTC m=+52.154086559 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls") pod "dns-default-gbnmx" (UID: "793d61b2-520e-416a-b856-545c0a0f45b5") : secret "dns-default-metrics-tls" not found
Jul 17 12:23:28.075834 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:28.075802    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls\") pod \"dns-default-gbnmx\" (UID: \"793d61b2-520e-416a-b856-545c0a0f45b5\") " pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:23:28.076329 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:28.075905    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:23:28.076329 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:28.075954    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls podName:793d61b2-520e-416a-b856-545c0a0f45b5 nodeName:}" failed. No retries permitted until 2026-07-17 12:23:44.075941144 +0000 UTC m=+68.207325784 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls") pod "dns-default-gbnmx" (UID: "793d61b2-520e-416a-b856-545c0a0f45b5") : secret "dns-default-metrics-tls" not found
Jul 17 12:23:38.711631 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:38.711601    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-trg5s"
Jul 17 12:23:41.164768 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:41.164723    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret\") pod \"global-pull-secret-syncer-vqfdd\" (UID: \"881c9321-df8f-45af-a283-2fe1306b33b5\") " pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:23:41.165188 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:41.164813    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs\") pod \"network-metrics-daemon-ml8cr\" (UID: \"e5f413a7-be80-43f3-950e-be9b5c4bf9b4\") " pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:23:41.167422 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:41.167402    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 12:23:41.167472 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:41.167402    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 12:23:41.175697 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:41.175681    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 17 12:23:41.175750 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:41.175730    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs podName:e5f413a7-be80-43f3-950e-be9b5c4bf9b4 nodeName:}" failed. No retries permitted until 2026-07-17 12:24:45.175714646 +0000 UTC m=+129.307099284 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs") pod "network-metrics-daemon-ml8cr" (UID: "e5f413a7-be80-43f3-950e-be9b5c4bf9b4") : secret "metrics-daemon-secret" not found
Jul 17 12:23:41.179241 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:41.179223    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/881c9321-df8f-45af-a283-2fe1306b33b5-original-pull-secret\") pod \"global-pull-secret-syncer-vqfdd\" (UID: \"881c9321-df8f-45af-a283-2fe1306b33b5\") " pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:23:41.265032 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:41.265001    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbpv8\" (UniqueName: \"kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8\") pod \"network-check-target-v89v8\" (UID: \"14587634-2d1d-472d-b470-981f1bc0ac8c\") " pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:23:41.267613 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:41.267598    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 12:23:41.277668 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:41.277652    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 12:23:41.287743 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:41.287723    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbpv8\" (UniqueName: \"kubernetes.io/projected/14587634-2d1d-472d-b470-981f1bc0ac8c-kube-api-access-hbpv8\") pod \"network-check-target-v89v8\" (UID: \"14587634-2d1d-472d-b470-981f1bc0ac8c\") " pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:23:41.310810 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:41.310792    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vqfdd"
Jul 17 12:23:41.426352 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:41.426288    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vqfdd"]
Jul 17 12:23:41.508281 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:41.508262    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2chv5\""
Jul 17 12:23:41.516551 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:41.516535    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:23:41.643712 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:41.643684    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v89v8"]
Jul 17 12:23:41.646561 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:23:41.646537    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14587634_2d1d_472d_b470_981f1bc0ac8c.slice/crio-e93077a7f8f1a526d59dd0c25146e48c2d6271b0ff9211c85ce0f05693785757 WatchSource:0}: Error finding container e93077a7f8f1a526d59dd0c25146e48c2d6271b0ff9211c85ce0f05693785757: Status 404 returned error can't find the container with id e93077a7f8f1a526d59dd0c25146e48c2d6271b0ff9211c85ce0f05693785757
Jul 17 12:23:41.785500 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:41.785421    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v89v8" event={"ID":"14587634-2d1d-472d-b470-981f1bc0ac8c","Type":"ContainerStarted","Data":"e93077a7f8f1a526d59dd0c25146e48c2d6271b0ff9211c85ce0f05693785757"}
Jul 17 12:23:41.786366 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:41.786344    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vqfdd" event={"ID":"881c9321-df8f-45af-a283-2fe1306b33b5","Type":"ContainerStarted","Data":"cc4a11b4ae74af42d6baaa09517ac325fe8370090216be4dcd0dd4bab8ab2363"}
Jul 17 12:23:44.086533 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:44.086500    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls\") pod \"dns-default-gbnmx\" (UID: \"793d61b2-520e-416a-b856-545c0a0f45b5\") " pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:23:44.086951 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:44.086671    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:23:44.086951 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:23:44.086753    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls podName:793d61b2-520e-416a-b856-545c0a0f45b5 nodeName:}" failed. No retries permitted until 2026-07-17 12:24:16.086734922 +0000 UTC m=+100.218119562 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls") pod "dns-default-gbnmx" (UID: "793d61b2-520e-416a-b856-545c0a0f45b5") : secret "dns-default-metrics-tls" not found
Jul 17 12:23:46.800732 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:46.800692    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vqfdd" event={"ID":"881c9321-df8f-45af-a283-2fe1306b33b5","Type":"ContainerStarted","Data":"eec094bda88ad0439c4bcc9689e200f24361581bbbe3fdf51fcca4fc8c1ae4f2"}
Jul 17 12:23:46.802096 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:46.802070    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v89v8" event={"ID":"14587634-2d1d-472d-b470-981f1bc0ac8c","Type":"ContainerStarted","Data":"ff2526889a26e0d0190ddf4fafbf84074c228f407bdab060131f55ceb18fe3d9"}
Jul 17 12:23:46.802202 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:46.802190    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:23:46.816142 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:46.816102    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vqfdd" podStartSLOduration=64.928253407 podStartE2EDuration="1m9.81608908s" totalImagesPullingTime="4.887835673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:23:46.815092595 +0000 UTC m=+70.946477254" watchObservedRunningTime="2026-07-17 12:23:46.81608908 +0000 UTC m=+70.947473739"
Jul 17 12:23:46.829531 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:23:46.829491    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-v89v8" podStartSLOduration=65.169954548 podStartE2EDuration="1m9.829479242s" totalImagesPullingTime="4.659524694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:23:46.828573086 +0000 UTC m=+70.959957743" watchObservedRunningTime="2026-07-17 12:23:46.829479242 +0000 UTC m=+70.960863908"
Jul 17 12:24:16.186509 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:24:16.186394    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls\") pod \"dns-default-gbnmx\" (UID: \"793d61b2-520e-416a-b856-545c0a0f45b5\") " pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:24:16.186949 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:24:16.186511    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:24:16.186949 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:24:16.186569    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls podName:793d61b2-520e-416a-b856-545c0a0f45b5 nodeName:}" failed. No retries permitted until 2026-07-17 12:25:20.186556997 +0000 UTC m=+164.317941635 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls") pod "dns-default-gbnmx" (UID: "793d61b2-520e-416a-b856-545c0a0f45b5") : secret "dns-default-metrics-tls" not found
Jul 17 12:24:17.806614 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:24:17.806586    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-v89v8"
Jul 17 12:24:27.170036 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:24:27.170006    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:24:45.268539 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:24:45.268498    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs\") pod \"network-metrics-daemon-ml8cr\" (UID: \"e5f413a7-be80-43f3-950e-be9b5c4bf9b4\") " pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:24:45.268903 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:24:45.268624    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 17 12:24:45.268903 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:24:45.268682    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs podName:e5f413a7-be80-43f3-950e-be9b5c4bf9b4 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:47.268664181 +0000 UTC m=+251.400048838 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs") pod "network-metrics-daemon-ml8cr" (UID: "e5f413a7-be80-43f3-950e-be9b5c4bf9b4") : secret "metrics-daemon-secret" not found
Jul 17 12:25:15.273524 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:25:15.273481    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-gbnmx" podUID="793d61b2-520e-416a-b856-545c0a0f45b5"
Jul 17 12:25:15.999026 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:25:15.998999    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:25:16.417681 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:25:16.417653    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-ml8cr" podUID="e5f413a7-be80-43f3-950e-be9b5c4bf9b4"
Jul 17 12:25:20.282247 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:25:20.282213    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls\") pod \"dns-default-gbnmx\" (UID: \"793d61b2-520e-416a-b856-545c0a0f45b5\") " pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:25:20.282598 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:25:20.282325    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 12:25:20.282598 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:25:20.282373    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls podName:793d61b2-520e-416a-b856-545c0a0f45b5 nodeName:}" failed. No retries permitted until 2026-07-17 12:27:22.282360424 +0000 UTC m=+286.413745063 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls") pod "dns-default-gbnmx" (UID: "793d61b2-520e-416a-b856-545c0a0f45b5") : secret "dns-default-metrics-tls" not found
Jul 17 12:25:24.512342 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:25:24.512308    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:25:29.396816 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:25:29.396726    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:26:26.958436 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:26.958402    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-4lhxw"]
Jul 17 12:26:26.960147 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:26.960129    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-4lhxw"
Jul 17 12:26:26.963327 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:26.963306    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-6829f\""
Jul 17 12:26:26.983329 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:26.983308    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-4lhxw"]
Jul 17 12:26:27.056401 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.056375    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-qn4xk"]
Jul 17 12:26:27.058122 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.058107    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-qn4xk"
Jul 17 12:26:27.061361 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.061332    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 17 12:26:27.061748 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.061728    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-698j8\""
Jul 17 12:26:27.061748 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.061738    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 17 12:26:27.062249 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.062228    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-p8vsw"]
Jul 17 12:26:27.064234 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.064209    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2mbv7"]
Jul 17 12:26:27.064740 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.064721    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-p8vsw"
Jul 17 12:26:27.065946 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.065933    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2mbv7"
Jul 17 12:26:27.067284 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.067266    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 17 12:26:27.067360 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.067326    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-k2p96\""
Jul 17 12:26:27.067610 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.067592    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 17 12:26:27.068477 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.068458    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 17 12:26:27.068574 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.068554    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 17 12:26:27.068777 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.068760    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 17 12:26:27.069809 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.069793    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-gcvxg\""
Jul 17 12:26:27.069890 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.069813    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 17 12:26:27.070180 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.070164    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 17 12:26:27.070449 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.070429    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 17 12:26:27.074320 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.074302    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 17 12:26:27.076053 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.076035    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-qn4xk"]
Jul 17 12:26:27.080578 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.080561    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2mbv7"]
Jul 17 12:26:27.081448 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.081430    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-p8vsw"]
Jul 17 12:26:27.093725 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.093704    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrmvw\" (UniqueName: \"kubernetes.io/projected/3643012f-6047-4c5c-a04a-cf1c801c7d11-kube-api-access-qrmvw\") pod \"network-check-source-7bc5b6bf85-4lhxw\" (UID: \"3643012f-6047-4c5c-a04a-cf1c801c7d11\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-4lhxw"
Jul 17 12:26:27.155903 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.155877    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6fcd94dc48-8sf54"]
Jul 17 12:26:27.157891 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.157872    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.160504 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.160490    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5tvcb\""
Jul 17 12:26:27.160768 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.160752    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 17 12:26:27.160833 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.160779    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 17 12:26:27.160994 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.160968    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 17 12:26:27.170933 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.170906    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 17 12:26:27.173128 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.173092    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fcd94dc48-8sf54"]
Jul 17 12:26:27.194297 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.194268    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-qn4xk\" (UID: \"2ea9caca-0266-4894-ad2e-c02b4e5eeddf\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-qn4xk"
Jul 17 12:26:27.194297 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.194296    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l7wj\" (UniqueName: \"kubernetes.io/projected/3b0956d7-a5e7-4710-8063-dbc0e5429f9f-kube-api-access-4l7wj\") pod \"console-operator-565b7b569d-p8vsw\" (UID: \"3b0956d7-a5e7-4710-8063-dbc0e5429f9f\") " pod="openshift-console-operator/console-operator-565b7b569d-p8vsw"
Jul 17 12:26:27.194449 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.194317    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b0956d7-a5e7-4710-8063-dbc0e5429f9f-serving-cert\") pod \"console-operator-565b7b569d-p8vsw\" (UID: \"3b0956d7-a5e7-4710-8063-dbc0e5429f9f\") " pod="openshift-console-operator/console-operator-565b7b569d-p8vsw"
Jul 17 12:26:27.194449 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.194369    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4b08097-9f28-46c0-95c0-404d419f9cac-serving-cert\") pod \"service-ca-operator-5bcbc94c78-2mbv7\" (UID: \"b4b08097-9f28-46c0-95c0-404d419f9cac\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2mbv7"
Jul 17 12:26:27.194449 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.194404    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b0956d7-a5e7-4710-8063-dbc0e5429f9f-trusted-ca\") pod \"console-operator-565b7b569d-p8vsw\" (UID: \"3b0956d7-a5e7-4710-8063-dbc0e5429f9f\") " pod="openshift-console-operator/console-operator-565b7b569d-p8vsw"
Jul 17 12:26:27.194449 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.194436    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4b08097-9f28-46c0-95c0-404d419f9cac-config\") pod \"service-ca-operator-5bcbc94c78-2mbv7\" (UID: \"b4b08097-9f28-46c0-95c0-404d419f9cac\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2mbv7"
Jul 17 12:26:27.194603 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.194461    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-qn4xk\" (UID: \"2ea9caca-0266-4894-ad2e-c02b4e5eeddf\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-qn4xk"
Jul 17 12:26:27.194603 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.194497    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb8dd\" (UniqueName: \"kubernetes.io/projected/b4b08097-9f28-46c0-95c0-404d419f9cac-kube-api-access-pb8dd\") pod \"service-ca-operator-5bcbc94c78-2mbv7\" (UID: \"b4b08097-9f28-46c0-95c0-404d419f9cac\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2mbv7"
Jul 17 12:26:27.194603 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.194521    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrmvw\" (UniqueName: \"kubernetes.io/projected/3643012f-6047-4c5c-a04a-cf1c801c7d11-kube-api-access-qrmvw\") pod \"network-check-source-7bc5b6bf85-4lhxw\" (UID: \"3643012f-6047-4c5c-a04a-cf1c801c7d11\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-4lhxw"
Jul 17 12:26:27.194603 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.194536    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3b0956d7-a5e7-4710-8063-dbc0e5429f9f-config\") pod \"console-operator-565b7b569d-p8vsw\" (UID: \"3b0956d7-a5e7-4710-8063-dbc0e5429f9f\") " pod="openshift-console-operator/console-operator-565b7b569d-p8vsw"
Jul 17 12:26:27.203050 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.203033    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrmvw\" (UniqueName: \"kubernetes.io/projected/3643012f-6047-4c5c-a04a-cf1c801c7d11-kube-api-access-qrmvw\") pod \"network-check-source-7bc5b6bf85-4lhxw\" (UID: \"3643012f-6047-4c5c-a04a-cf1c801c7d11\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-4lhxw"
Jul 17 12:26:27.268154 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.268098    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-4lhxw"
Jul 17 12:26:27.295034 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.295007    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-qn4xk\" (UID: \"2ea9caca-0266-4894-ad2e-c02b4e5eeddf\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-qn4xk"
Jul 17 12:26:27.295136 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.295062    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-tls\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.295136 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.295089    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-ca-trust-extracted\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.295236 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.295142    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l7wj\" (UniqueName: \"kubernetes.io/projected/3b0956d7-a5e7-4710-8063-dbc0e5429f9f-kube-api-access-4l7wj\") pod \"console-operator-565b7b569d-p8vsw\" (UID: \"3b0956d7-a5e7-4710-8063-dbc0e5429f9f\") " pod="openshift-console-operator/console-operator-565b7b569d-p8vsw"
Jul 17 12:26:27.295236 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.295167    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-trusted-ca\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.295236 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.295191    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-installation-pull-secrets\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.295364 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.295255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b0956d7-a5e7-4710-8063-dbc0e5429f9f-serving-cert\") pod \"console-operator-565b7b569d-p8vsw\" (UID: \"3b0956d7-a5e7-4710-8063-dbc0e5429f9f\") " pod="openshift-console-operator/console-operator-565b7b569d-p8vsw"
Jul 17 12:26:27.295364 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.295304    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-image-registry-private-configuration\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.295364 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.295345    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4b08097-9f28-46c0-95c0-404d419f9cac-serving-cert\") pod \"service-ca-operator-5bcbc94c78-2mbv7\" (UID: \"b4b08097-9f28-46c0-95c0-404d419f9cac\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2mbv7"
Jul 17 12:26:27.295493 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.295377    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-bound-sa-token\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.295493 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.295412    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b0956d7-a5e7-4710-8063-dbc0e5429f9f-trusted-ca\") pod \"console-operator-565b7b569d-p8vsw\" (UID: \"3b0956d7-a5e7-4710-8063-dbc0e5429f9f\") " pod="openshift-console-operator/console-operator-565b7b569d-p8vsw"
Jul 17 12:26:27.295493 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.295465    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4b08097-9f28-46c0-95c0-404d419f9cac-config\") pod \"service-ca-operator-5bcbc94c78-2mbv7\" (UID: \"b4b08097-9f28-46c0-95c0-404d419f9cac\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2mbv7"
Jul 17 12:26:27.295641 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.295502    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-qn4xk\" (UID: \"2ea9caca-0266-4894-ad2e-c02b4e5eeddf\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-qn4xk"
Jul 17 12:26:27.295641 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.295544    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-qn4xk\" (UID: \"2ea9caca-0266-4894-ad2e-c02b4e5eeddf\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-qn4xk"
Jul 17 12:26:27.295641 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.295542    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb8dd\" (UniqueName: \"kubernetes.io/projected/b4b08097-9f28-46c0-95c0-404d419f9cac-kube-api-access-pb8dd\") pod \"service-ca-operator-5bcbc94c78-2mbv7\" (UID: \"b4b08097-9f28-46c0-95c0-404d419f9cac\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2mbv7"
Jul 17 12:26:27.295641 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.295609    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3b0956d7-a5e7-4710-8063-dbc0e5429f9f-config\") pod \"console-operator-565b7b569d-p8vsw\" (UID: \"3b0956d7-a5e7-4710-8063-dbc0e5429f9f\") " pod="openshift-console-operator/console-operator-565b7b569d-p8vsw"
Jul 17 12:26:27.295641 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:27.295638    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 12:26:27.295869 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.295638    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t74bc\" (UniqueName: \"kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-kube-api-access-t74bc\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.295869 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.295687    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-certificates\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.295869 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:27.295700    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-networking-console-plugin-cert podName:2ea9caca-0266-4894-ad2e-c02b4e5eeddf nodeName:}" failed. No retries permitted until 2026-07-17 12:26:27.795681582 +0000 UTC m=+231.927066235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-qn4xk" (UID: "2ea9caca-0266-4894-ad2e-c02b4e5eeddf") : secret "networking-console-plugin-cert" not found
Jul 17 12:26:27.296090 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.296066    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4b08097-9f28-46c0-95c0-404d419f9cac-config\") pod \"service-ca-operator-5bcbc94c78-2mbv7\" (UID: \"b4b08097-9f28-46c0-95c0-404d419f9cac\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2mbv7"
Jul 17 12:26:27.296415 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.296396    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b0956d7-a5e7-4710-8063-dbc0e5429f9f-trusted-ca\") pod \"console-operator-565b7b569d-p8vsw\" (UID: \"3b0956d7-a5e7-4710-8063-dbc0e5429f9f\") " pod="openshift-console-operator/console-operator-565b7b569d-p8vsw"
Jul 17 12:26:27.296581 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.296559    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3b0956d7-a5e7-4710-8063-dbc0e5429f9f-config\") pod \"console-operator-565b7b569d-p8vsw\" (UID: \"3b0956d7-a5e7-4710-8063-dbc0e5429f9f\") " pod="openshift-console-operator/console-operator-565b7b569d-p8vsw"
Jul 17 12:26:27.297795 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.297773    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b0956d7-a5e7-4710-8063-dbc0e5429f9f-serving-cert\") pod \"console-operator-565b7b569d-p8vsw\" (UID: \"3b0956d7-a5e7-4710-8063-dbc0e5429f9f\") " pod="openshift-console-operator/console-operator-565b7b569d-p8vsw"
Jul 17 12:26:27.298016 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.297975    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4b08097-9f28-46c0-95c0-404d419f9cac-serving-cert\") pod \"service-ca-operator-5bcbc94c78-2mbv7\" (UID: \"b4b08097-9f28-46c0-95c0-404d419f9cac\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2mbv7"
Jul 17 12:26:27.306451 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.306432    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb8dd\" (UniqueName: \"kubernetes.io/projected/b4b08097-9f28-46c0-95c0-404d419f9cac-kube-api-access-pb8dd\") pod \"service-ca-operator-5bcbc94c78-2mbv7\" (UID: \"b4b08097-9f28-46c0-95c0-404d419f9cac\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2mbv7"
Jul 17 12:26:27.306529 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.306494    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l7wj\" (UniqueName: \"kubernetes.io/projected/3b0956d7-a5e7-4710-8063-dbc0e5429f9f-kube-api-access-4l7wj\") pod \"console-operator-565b7b569d-p8vsw\" (UID: \"3b0956d7-a5e7-4710-8063-dbc0e5429f9f\") " pod="openshift-console-operator/console-operator-565b7b569d-p8vsw"
Jul 17 12:26:27.376507 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.376474    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-p8vsw"
Jul 17 12:26:27.382138 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.382115    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2mbv7"
Jul 17 12:26:27.385066 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.385024    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-4lhxw"]
Jul 17 12:26:27.388413 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:26:27.388386    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3643012f_6047_4c5c_a04a_cf1c801c7d11.slice/crio-9acafce5c51d969d9f8e640cf0c2806acfe90c7fbc9a5f02169ec31f0a396f01 WatchSource:0}: Error finding container 9acafce5c51d969d9f8e640cf0c2806acfe90c7fbc9a5f02169ec31f0a396f01: Status 404 returned error can't find the container with id 9acafce5c51d969d9f8e640cf0c2806acfe90c7fbc9a5f02169ec31f0a396f01
Jul 17 12:26:27.397051 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.397027    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-trusted-ca\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.397150 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.397064    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-installation-pull-secrets\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.397150 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.397127    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-image-registry-private-configuration\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.397417 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.397160    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-bound-sa-token\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.397417 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.397210    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t74bc\" (UniqueName: \"kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-kube-api-access-t74bc\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.397417 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.397239    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-certificates\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.397417 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.397277    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-tls\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.397417 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.397300    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-ca-trust-extracted\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.398210 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:27.397838    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 12:26:27.398210 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:27.397859    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcd94dc48-8sf54: secret "image-registry-tls" not found
Jul 17 12:26:27.398210 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:27.397911    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-tls podName:f8864247-35de-4e6c-bfb3-0a97f1f2aabf nodeName:}" failed. No retries permitted until 2026-07-17 12:26:27.897893546 +0000 UTC m=+232.029278203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-tls") pod "image-registry-6fcd94dc48-8sf54" (UID: "f8864247-35de-4e6c-bfb3-0a97f1f2aabf") : secret "image-registry-tls" not found
Jul 17 12:26:27.398210 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.397910    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-ca-trust-extracted\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.398210 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.398059    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-trusted-ca\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.398535 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.398335    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-certificates\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.399925 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.399904    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-image-registry-private-configuration\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.400337 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.400319    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-installation-pull-secrets\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.406060 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.406038    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-bound-sa-token\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.407724 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.407700    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t74bc\" (UniqueName: \"kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-kube-api-access-t74bc\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.513857 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.513823    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-p8vsw"]
Jul 17 12:26:27.516245 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:26:27.516209    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0956d7_a5e7_4710_8063_dbc0e5429f9f.slice/crio-a42361cbfa43767feffc77bd02e32c660840e6526de12f5626d5c899038bce94 WatchSource:0}: Error finding container a42361cbfa43767feffc77bd02e32c660840e6526de12f5626d5c899038bce94: Status 404 returned error can't find the container with id a42361cbfa43767feffc77bd02e32c660840e6526de12f5626d5c899038bce94
Jul 17 12:26:27.523611 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.523593    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2mbv7"]
Jul 17 12:26:27.526002 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:26:27.525965    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b08097_9f28_46c0_95c0_404d419f9cac.slice/crio-50b90d43d2cf6f6684b1a9d18f542be6f481a1890f32fb29be192b337511d332 WatchSource:0}: Error finding container 50b90d43d2cf6f6684b1a9d18f542be6f481a1890f32fb29be192b337511d332: Status 404 returned error can't find the container with id 50b90d43d2cf6f6684b1a9d18f542be6f481a1890f32fb29be192b337511d332
Jul 17 12:26:27.800510 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.800478    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-qn4xk\" (UID: \"2ea9caca-0266-4894-ad2e-c02b4e5eeddf\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-qn4xk"
Jul 17 12:26:27.800638 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:27.800617    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 12:26:27.800679 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:27.800673    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-networking-console-plugin-cert podName:2ea9caca-0266-4894-ad2e-c02b4e5eeddf nodeName:}" failed. No retries permitted until 2026-07-17 12:26:28.800658987 +0000 UTC m=+232.932043625 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-qn4xk" (UID: "2ea9caca-0266-4894-ad2e-c02b4e5eeddf") : secret "networking-console-plugin-cert" not found
Jul 17 12:26:27.901832 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:27.901804    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-tls\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:27.901938 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:27.901910    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 12:26:27.901938 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:27.901920    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcd94dc48-8sf54: secret "image-registry-tls" not found
Jul 17 12:26:27.902041 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:27.901960    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-tls podName:f8864247-35de-4e6c-bfb3-0a97f1f2aabf nodeName:}" failed. No retries permitted until 2026-07-17 12:26:28.901948664 +0000 UTC m=+233.033333303 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-tls") pod "image-registry-6fcd94dc48-8sf54" (UID: "f8864247-35de-4e6c-bfb3-0a97f1f2aabf") : secret "image-registry-tls" not found
Jul 17 12:26:28.164432 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:28.164349    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2mbv7" event={"ID":"b4b08097-9f28-46c0-95c0-404d419f9cac","Type":"ContainerStarted","Data":"50b90d43d2cf6f6684b1a9d18f542be6f481a1890f32fb29be192b337511d332"}
Jul 17 12:26:28.166180 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:28.166153    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-4lhxw" event={"ID":"3643012f-6047-4c5c-a04a-cf1c801c7d11","Type":"ContainerStarted","Data":"26d4e79138318a1ebead6603b0882ac75219e684df72591243d2b283a4b957c9"}
Jul 17 12:26:28.166298 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:28.166233    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-4lhxw" event={"ID":"3643012f-6047-4c5c-a04a-cf1c801c7d11","Type":"ContainerStarted","Data":"9acafce5c51d969d9f8e640cf0c2806acfe90c7fbc9a5f02169ec31f0a396f01"}
Jul 17 12:26:28.167500 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:28.167460    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-p8vsw" event={"ID":"3b0956d7-a5e7-4710-8063-dbc0e5429f9f","Type":"ContainerStarted","Data":"a42361cbfa43767feffc77bd02e32c660840e6526de12f5626d5c899038bce94"}
Jul 17 12:26:28.181995 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:28.181936    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-4lhxw" podStartSLOduration=2.181921484 podStartE2EDuration="2.181921484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:26:28.181331997 +0000 UTC m=+232.312716653" watchObservedRunningTime="2026-07-17 12:26:28.181921484 +0000 UTC m=+232.313306183"
Jul 17 12:26:28.809356 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:28.809322    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-qn4xk\" (UID: \"2ea9caca-0266-4894-ad2e-c02b4e5eeddf\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-qn4xk"
Jul 17 12:26:28.809547 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:28.809496    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 12:26:28.809597 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:28.809579    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-networking-console-plugin-cert podName:2ea9caca-0266-4894-ad2e-c02b4e5eeddf nodeName:}" failed. No retries permitted until 2026-07-17 12:26:30.8095585 +0000 UTC m=+234.940943149 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-qn4xk" (UID: "2ea9caca-0266-4894-ad2e-c02b4e5eeddf") : secret "networking-console-plugin-cert" not found
Jul 17 12:26:28.909902 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:28.909868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-tls\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:28.910130 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:28.910103    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 12:26:28.910201 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:28.910137    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcd94dc48-8sf54: secret "image-registry-tls" not found
Jul 17 12:26:28.910240 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:28.910224    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-tls podName:f8864247-35de-4e6c-bfb3-0a97f1f2aabf nodeName:}" failed. No retries permitted until 2026-07-17 12:26:30.910200546 +0000 UTC m=+235.041585228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-tls") pod "image-registry-6fcd94dc48-8sf54" (UID: "f8864247-35de-4e6c-bfb3-0a97f1f2aabf") : secret "image-registry-tls" not found
Jul 17 12:26:30.174160 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:30.174123    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2mbv7" event={"ID":"b4b08097-9f28-46c0-95c0-404d419f9cac","Type":"ContainerStarted","Data":"75a953442ed7fa5934475a1fb37fa954c7420e358256a24c8717da6f5023acee"}
Jul 17 12:26:30.175722 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:30.175705    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/0.log"
Jul 17 12:26:30.175812 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:30.175738    2565 generic.go:356] "Generic (PLEG): container finished" podID="3b0956d7-a5e7-4710-8063-dbc0e5429f9f" containerID="a1f72ad77524c0bdc690cdc8e8d3538a01e7344127dc2f483862f2c7da4ec9b8" exitCode=255
Jul 17 12:26:30.175812 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:30.175787    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-p8vsw" event={"ID":"3b0956d7-a5e7-4710-8063-dbc0e5429f9f","Type":"ContainerDied","Data":"a1f72ad77524c0bdc690cdc8e8d3538a01e7344127dc2f483862f2c7da4ec9b8"}
Jul 17 12:26:30.175962 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:30.175951    2565 scope.go:117] "RemoveContainer" containerID="a1f72ad77524c0bdc690cdc8e8d3538a01e7344127dc2f483862f2c7da4ec9b8"
Jul 17 12:26:30.192365 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:30.192325    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2mbv7" podStartSLOduration=0.731749931 podStartE2EDuration="3.192311378s" totalImagesPullingTime="2.460561447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:26:30.191352023 +0000 UTC m=+234.322736687" watchObservedRunningTime="2026-07-17 12:26:30.192311378 +0000 UTC m=+234.323696016"
Jul 17 12:26:30.797251 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:30.797222    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8qwp2"]
Jul 17 12:26:30.801889 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:30.801863    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:30.804946 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:30.804916    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 17 12:26:30.805563 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:30.805546    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 17 12:26:30.805816 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:30.805792    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nmxgd\""
Jul 17 12:26:30.805886 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:30.805827    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 17 12:26:30.806031 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:30.806019    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 17 12:26:30.817176 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:30.817156    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8qwp2"]
Jul 17 12:26:30.826480 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:30.826458    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-qn4xk\" (UID: \"2ea9caca-0266-4894-ad2e-c02b4e5eeddf\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-qn4xk"
Jul 17 12:26:30.826604 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:30.826587    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 12:26:30.826662 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:30.826644    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-networking-console-plugin-cert podName:2ea9caca-0266-4894-ad2e-c02b4e5eeddf nodeName:}" failed. No retries permitted until 2026-07-17 12:26:34.826629401 +0000 UTC m=+238.958014039 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-qn4xk" (UID: "2ea9caca-0266-4894-ad2e-c02b4e5eeddf") : secret "networking-console-plugin-cert" not found
Jul 17 12:26:30.927345 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:30.927310    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-tls\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:30.927513 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:30.927352    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkgpv\" (UniqueName: \"kubernetes.io/projected/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-kube-api-access-xkgpv\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:30.927513 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:30.927392    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:30.927513 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:30.927410    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:30.927513 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:30.927431    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-crio-socket\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:30.927513 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:30.927460    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 12:26:30.927513 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:30.927479    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcd94dc48-8sf54: secret "image-registry-tls" not found
Jul 17 12:26:30.927513 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:30.927468    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-data-volume\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:30.927773 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:30.927550    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-tls podName:f8864247-35de-4e6c-bfb3-0a97f1f2aabf nodeName:}" failed. No retries permitted until 2026-07-17 12:26:34.927533348 +0000 UTC m=+239.058917986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-tls") pod "image-registry-6fcd94dc48-8sf54" (UID: "f8864247-35de-4e6c-bfb3-0a97f1f2aabf") : secret "image-registry-tls" not found
Jul 17 12:26:31.029002 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:31.028953    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkgpv\" (UniqueName: \"kubernetes.io/projected/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-kube-api-access-xkgpv\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:31.029158 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:31.029016    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:31.029158 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:31.029063    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:31.029158 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:31.029083    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-crio-socket\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:31.029158 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:31.029114    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-data-volume\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:31.029374 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:31.029170    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 12:26:31.029374 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:31.029200    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-crio-socket\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:31.029374 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:31.029205    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 12:26:31.029374 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:31.029263    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-kube-rbac-proxy-cm podName:97f2ceb2-b1ef-4380-b3ce-8e9ff2498282 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:31.529241551 +0000 UTC m=+235.660626190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8qwp2" (UID: "97f2ceb2-b1ef-4380-b3ce-8e9ff2498282") : configmap "kube-rbac-proxy" not found
Jul 17 12:26:31.029374 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:31.029284    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-insights-runtime-extractor-tls podName:97f2ceb2-b1ef-4380-b3ce-8e9ff2498282 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:31.529273808 +0000 UTC m=+235.660658452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8qwp2" (UID: "97f2ceb2-b1ef-4380-b3ce-8e9ff2498282") : secret "insights-runtime-extractor-tls" not found
Jul 17 12:26:31.029546 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:31.029444    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-data-volume\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:31.037252 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:31.037236    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkgpv\" (UniqueName: \"kubernetes.io/projected/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-kube-api-access-xkgpv\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:31.180343 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:31.180264    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 12:26:31.180694 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:31.180680    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/0.log"
Jul 17 12:26:31.180729 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:31.180713    2565 generic.go:356] "Generic (PLEG): container finished" podID="3b0956d7-a5e7-4710-8063-dbc0e5429f9f" containerID="c0923cdb87ab76413a5e62a20e7370d1770e83224f135350c44c737b947aac7f" exitCode=255
Jul 17 12:26:31.180800 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:31.180787    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-p8vsw" event={"ID":"3b0956d7-a5e7-4710-8063-dbc0e5429f9f","Type":"ContainerDied","Data":"c0923cdb87ab76413a5e62a20e7370d1770e83224f135350c44c737b947aac7f"}
Jul 17 12:26:31.180847 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:31.180839    2565 scope.go:117] "RemoveContainer" containerID="a1f72ad77524c0bdc690cdc8e8d3538a01e7344127dc2f483862f2c7da4ec9b8"
Jul 17 12:26:31.181078 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:31.181060    2565 scope.go:117] "RemoveContainer" containerID="c0923cdb87ab76413a5e62a20e7370d1770e83224f135350c44c737b947aac7f"
Jul 17 12:26:31.181263 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:31.181243    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-p8vsw_openshift-console-operator(3b0956d7-a5e7-4710-8063-dbc0e5429f9f)\"" pod="openshift-console-operator/console-operator-565b7b569d-p8vsw" podUID="3b0956d7-a5e7-4710-8063-dbc0e5429f9f"
Jul 17 12:26:31.532569 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:31.532499    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:31.532569 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:31.532530    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:31.532738 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:31.532623    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 12:26:31.532738 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:31.532677    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 12:26:31.532738 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:31.532694    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-kube-rbac-proxy-cm podName:97f2ceb2-b1ef-4380-b3ce-8e9ff2498282 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:32.532675006 +0000 UTC m=+236.664059656 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8qwp2" (UID: "97f2ceb2-b1ef-4380-b3ce-8e9ff2498282") : configmap "kube-rbac-proxy" not found
Jul 17 12:26:31.532738 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:31.532724    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-insights-runtime-extractor-tls podName:97f2ceb2-b1ef-4380-b3ce-8e9ff2498282 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:32.532713211 +0000 UTC m=+236.664097849 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8qwp2" (UID: "97f2ceb2-b1ef-4380-b3ce-8e9ff2498282") : secret "insights-runtime-extractor-tls" not found
Jul 17 12:26:32.184899 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:32.184875    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 12:26:32.185296 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:32.185181    2565 scope.go:117] "RemoveContainer" containerID="c0923cdb87ab76413a5e62a20e7370d1770e83224f135350c44c737b947aac7f"
Jul 17 12:26:32.185348 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:32.185307    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-p8vsw_openshift-console-operator(3b0956d7-a5e7-4710-8063-dbc0e5429f9f)\"" pod="openshift-console-operator/console-operator-565b7b569d-p8vsw" podUID="3b0956d7-a5e7-4710-8063-dbc0e5429f9f"
Jul 17 12:26:32.541182 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:32.541108    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:32.541182 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:32.541148    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:32.541336 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:32.541216    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 12:26:32.541336 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:32.541233    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 12:26:32.541336 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:32.541268    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-kube-rbac-proxy-cm podName:97f2ceb2-b1ef-4380-b3ce-8e9ff2498282 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:34.541255217 +0000 UTC m=+238.672639855 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8qwp2" (UID: "97f2ceb2-b1ef-4380-b3ce-8e9ff2498282") : configmap "kube-rbac-proxy" not found
Jul 17 12:26:32.541336 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:32.541280    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-insights-runtime-extractor-tls podName:97f2ceb2-b1ef-4380-b3ce-8e9ff2498282 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:34.541274757 +0000 UTC m=+238.672659399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8qwp2" (UID: "97f2ceb2-b1ef-4380-b3ce-8e9ff2498282") : secret "insights-runtime-extractor-tls" not found
Jul 17 12:26:32.760151 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:32.760126    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kw5vk_d0be7fb8-4dda-4b00-9b21-6c105e5bff69/dns-node-resolver/0.log"
Jul 17 12:26:33.559781 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:33.559760    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b25ws_de956422-3a00-4360-87d9-e43b2cb93859/node-ca/0.log"
Jul 17 12:26:34.556056 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:34.556029    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:34.556056 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:34.556059    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:34.556233 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:34.556145    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 12:26:34.556233 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:34.556157    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 12:26:34.556233 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:34.556200    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-kube-rbac-proxy-cm podName:97f2ceb2-b1ef-4380-b3ce-8e9ff2498282 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:38.556185718 +0000 UTC m=+242.687570356 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8qwp2" (UID: "97f2ceb2-b1ef-4380-b3ce-8e9ff2498282") : configmap "kube-rbac-proxy" not found
Jul 17 12:26:34.556233 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:34.556213    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-insights-runtime-extractor-tls podName:97f2ceb2-b1ef-4380-b3ce-8e9ff2498282 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:38.55620738 +0000 UTC m=+242.687592018 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8qwp2" (UID: "97f2ceb2-b1ef-4380-b3ce-8e9ff2498282") : secret "insights-runtime-extractor-tls" not found
Jul 17 12:26:34.560476 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:34.560457    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n7hp4_02906c20-b77c-4c91-b4e4-fcd72c647670/serve-healthcheck-canary/0.log"
Jul 17 12:26:34.858779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:34.858715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-qn4xk\" (UID: \"2ea9caca-0266-4894-ad2e-c02b4e5eeddf\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-qn4xk"
Jul 17 12:26:34.858882 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:34.858832    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 12:26:34.858882 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:34.858880    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-networking-console-plugin-cert podName:2ea9caca-0266-4894-ad2e-c02b4e5eeddf nodeName:}" failed. No retries permitted until 2026-07-17 12:26:42.858868456 +0000 UTC m=+246.990253094 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-qn4xk" (UID: "2ea9caca-0266-4894-ad2e-c02b4e5eeddf") : secret "networking-console-plugin-cert" not found
Jul 17 12:26:34.959786 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:34.959756    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-tls\") pod \"image-registry-6fcd94dc48-8sf54\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") " pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:34.959926 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:34.959910    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 12:26:34.960008 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:34.959929    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6fcd94dc48-8sf54: secret "image-registry-tls" not found
Jul 17 12:26:34.960065 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:34.960020    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-tls podName:f8864247-35de-4e6c-bfb3-0a97f1f2aabf nodeName:}" failed. No retries permitted until 2026-07-17 12:26:42.959978925 +0000 UTC m=+247.091363585 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-tls") pod "image-registry-6fcd94dc48-8sf54" (UID: "f8864247-35de-4e6c-bfb3-0a97f1f2aabf") : secret "image-registry-tls" not found
Jul 17 12:26:37.377380 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:37.377351    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-p8vsw"
Jul 17 12:26:37.377706 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:37.377399    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-p8vsw"
Jul 17 12:26:37.377706 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:37.377677    2565 scope.go:117] "RemoveContainer" containerID="c0923cdb87ab76413a5e62a20e7370d1770e83224f135350c44c737b947aac7f"
Jul 17 12:26:37.377822 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:37.377802    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-p8vsw_openshift-console-operator(3b0956d7-a5e7-4710-8063-dbc0e5429f9f)\"" pod="openshift-console-operator/console-operator-565b7b569d-p8vsw" podUID="3b0956d7-a5e7-4710-8063-dbc0e5429f9f"
Jul 17 12:26:38.586642 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:38.586615    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:38.587030 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:38.586646    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:38.587030 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:38.586732    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 12:26:38.587030 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:38.586763    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 12:26:38.587030 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:38.586786    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-kube-rbac-proxy-cm podName:97f2ceb2-b1ef-4380-b3ce-8e9ff2498282 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:46.586771928 +0000 UTC m=+250.718156566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8qwp2" (UID: "97f2ceb2-b1ef-4380-b3ce-8e9ff2498282") : configmap "kube-rbac-proxy" not found
Jul 17 12:26:38.587030 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:38.586804    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-insights-runtime-extractor-tls podName:97f2ceb2-b1ef-4380-b3ce-8e9ff2498282 nodeName:}" failed. No retries permitted until 2026-07-17 12:26:46.586794032 +0000 UTC m=+250.718178670 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8qwp2" (UID: "97f2ceb2-b1ef-4380-b3ce-8e9ff2498282") : secret "insights-runtime-extractor-tls" not found
Jul 17 12:26:40.733287 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.733255    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fcd94dc48-8sf54"]
Jul 17 12:26:40.733624 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:40.733417    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54" podUID="f8864247-35de-4e6c-bfb3-0a97f1f2aabf"
Jul 17 12:26:40.763686 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.763665    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67b89bcb4d-gsdzm"]
Jul 17 12:26:40.766583 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.766569    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.778454 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.778434    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67b89bcb4d-gsdzm"]
Jul 17 12:26:40.799794 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.799771    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f080f086-e76c-4839-baa2-e07d90fc2a80-registry-tls\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.799878 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.799830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f080f086-e76c-4839-baa2-e07d90fc2a80-bound-sa-token\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.799927 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.799902    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f080f086-e76c-4839-baa2-e07d90fc2a80-trusted-ca\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.799958 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.799941    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f080f086-e76c-4839-baa2-e07d90fc2a80-image-registry-private-configuration\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.800007 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.799962    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f080f086-e76c-4839-baa2-e07d90fc2a80-installation-pull-secrets\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.800007 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.800000    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28q6t\" (UniqueName: \"kubernetes.io/projected/f080f086-e76c-4839-baa2-e07d90fc2a80-kube-api-access-28q6t\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.800092 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.800032    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f080f086-e76c-4839-baa2-e07d90fc2a80-registry-certificates\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.800092 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.800055    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f080f086-e76c-4839-baa2-e07d90fc2a80-ca-trust-extracted\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.900365 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.900329    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f080f086-e76c-4839-baa2-e07d90fc2a80-ca-trust-extracted\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.900466 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.900384    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f080f086-e76c-4839-baa2-e07d90fc2a80-registry-tls\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.900528 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.900502    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f080f086-e76c-4839-baa2-e07d90fc2a80-bound-sa-token\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.900577 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.900548    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f080f086-e76c-4839-baa2-e07d90fc2a80-trusted-ca\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.900654 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.900627    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f080f086-e76c-4839-baa2-e07d90fc2a80-image-registry-private-configuration\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.900654 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.900647    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f080f086-e76c-4839-baa2-e07d90fc2a80-ca-trust-extracted\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.900775 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.900674    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f080f086-e76c-4839-baa2-e07d90fc2a80-installation-pull-secrets\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.900775 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.900700    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28q6t\" (UniqueName: \"kubernetes.io/projected/f080f086-e76c-4839-baa2-e07d90fc2a80-kube-api-access-28q6t\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.900775 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.900739    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f080f086-e76c-4839-baa2-e07d90fc2a80-registry-certificates\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.901266 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.901245    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f080f086-e76c-4839-baa2-e07d90fc2a80-registry-certificates\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.901402 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.901385    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f080f086-e76c-4839-baa2-e07d90fc2a80-trusted-ca\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.902943 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.902927    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f080f086-e76c-4839-baa2-e07d90fc2a80-image-registry-private-configuration\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.903061 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.903041    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f080f086-e76c-4839-baa2-e07d90fc2a80-registry-tls\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.903128 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.903114    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f080f086-e76c-4839-baa2-e07d90fc2a80-installation-pull-secrets\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.909486 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.909465    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f080f086-e76c-4839-baa2-e07d90fc2a80-bound-sa-token\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:40.909659 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:40.909640    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28q6t\" (UniqueName: \"kubernetes.io/projected/f080f086-e76c-4839-baa2-e07d90fc2a80-kube-api-access-28q6t\") pod \"image-registry-67b89bcb4d-gsdzm\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") " pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:41.077544 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.077521    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5tvcb\""
Jul 17 12:26:41.085527 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.085510    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:41.202953 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.202927    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67b89bcb4d-gsdzm"]
Jul 17 12:26:41.204870 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:26:41.204846    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf080f086_e76c_4839_baa2_e07d90fc2a80.slice/crio-51c18c1c1a045219cfe3bf750bbb401b4a1c90c855bb4eed2ea521088cda3e59 WatchSource:0}: Error finding container 51c18c1c1a045219cfe3bf750bbb401b4a1c90c855bb4eed2ea521088cda3e59: Status 404 returned error can't find the container with id 51c18c1c1a045219cfe3bf750bbb401b4a1c90c855bb4eed2ea521088cda3e59
Jul 17 12:26:41.212197 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.212176    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm" event={"ID":"f080f086-e76c-4839-baa2-e07d90fc2a80","Type":"ContainerStarted","Data":"51c18c1c1a045219cfe3bf750bbb401b4a1c90c855bb4eed2ea521088cda3e59"}
Jul 17 12:26:41.212270 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.212205    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:41.217650 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.217634    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:41.304162 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.304136    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-bound-sa-token\") pod \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") "
Jul 17 12:26:41.304287 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.304244    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-certificates\") pod \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") "
Jul 17 12:26:41.304287 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.304283    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t74bc\" (UniqueName: \"kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-kube-api-access-t74bc\") pod \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") "
Jul 17 12:26:41.304389 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.304314    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-trusted-ca\") pod \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") "
Jul 17 12:26:41.304389 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.304371    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-ca-trust-extracted\") pod \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") "
Jul 17 12:26:41.304487 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.304399    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-image-registry-private-configuration\") pod \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") "
Jul 17 12:26:41.304687 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.304617    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f8864247-35de-4e6c-bfb3-0a97f1f2aabf" (UID: "f8864247-35de-4e6c-bfb3-0a97f1f2aabf"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:26:41.304687 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.304617    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f8864247-35de-4e6c-bfb3-0a97f1f2aabf" (UID: "f8864247-35de-4e6c-bfb3-0a97f1f2aabf"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:26:41.304999 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.304956    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f8864247-35de-4e6c-bfb3-0a97f1f2aabf" (UID: "f8864247-35de-4e6c-bfb3-0a97f1f2aabf"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:26:41.306137 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.306118    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f8864247-35de-4e6c-bfb3-0a97f1f2aabf" (UID: "f8864247-35de-4e6c-bfb3-0a97f1f2aabf"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:26:41.306364 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.306342    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f8864247-35de-4e6c-bfb3-0a97f1f2aabf" (UID: "f8864247-35de-4e6c-bfb3-0a97f1f2aabf"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:26:41.306520 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.306504    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-kube-api-access-t74bc" (OuterVolumeSpecName: "kube-api-access-t74bc") pod "f8864247-35de-4e6c-bfb3-0a97f1f2aabf" (UID: "f8864247-35de-4e6c-bfb3-0a97f1f2aabf"). InnerVolumeSpecName "kube-api-access-t74bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:26:41.405730 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.405660    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-installation-pull-secrets\") pod \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\" (UID: \"f8864247-35de-4e6c-bfb3-0a97f1f2aabf\") "
Jul 17 12:26:41.405854 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.405796    2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-ca-trust-extracted\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:26:41.405854 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.405808    2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-image-registry-private-configuration\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:26:41.405854 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.405817    2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-bound-sa-token\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:26:41.405854 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.405826    2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-certificates\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:26:41.405854 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.405834    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t74bc\" (UniqueName: \"kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-kube-api-access-t74bc\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:26:41.405854 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.405842    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-trusted-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:26:41.407561 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.407541    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f8864247-35de-4e6c-bfb3-0a97f1f2aabf" (UID: "f8864247-35de-4e6c-bfb3-0a97f1f2aabf"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:26:41.507082 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:41.507051    2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-installation-pull-secrets\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:26:42.217193 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:42.217125    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fcd94dc48-8sf54"
Jul 17 12:26:42.217193 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:42.217159    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm" event={"ID":"f080f086-e76c-4839-baa2-e07d90fc2a80","Type":"ContainerStarted","Data":"69c34abe2911d4e75183a1c7223892dac51a7649d422302cac63da8a9a69dc88"}
Jul 17 12:26:42.217660 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:42.217372    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:26:42.236491 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:42.236455    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm" podStartSLOduration=2.236442921 podStartE2EDuration="2.236442921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:26:42.235531986 +0000 UTC m=+246.366916670" watchObservedRunningTime="2026-07-17 12:26:42.236442921 +0000 UTC m=+246.367827579"
Jul 17 12:26:42.265842 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:42.265815    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6fcd94dc48-8sf54"]
Jul 17 12:26:42.269204 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:42.269178    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6fcd94dc48-8sf54"]
Jul 17 12:26:42.312987 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:42.312962    2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8864247-35de-4e6c-bfb3-0a97f1f2aabf-registry-tls\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:26:42.400097 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:42.400075    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8864247-35de-4e6c-bfb3-0a97f1f2aabf" path="/var/lib/kubelet/pods/f8864247-35de-4e6c-bfb3-0a97f1f2aabf/volumes"
Jul 17 12:26:42.918325 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:42.918297    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-qn4xk\" (UID: \"2ea9caca-0266-4894-ad2e-c02b4e5eeddf\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-qn4xk"
Jul 17 12:26:42.918443 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:42.918389    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 12:26:42.918443 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:26:42.918438    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-networking-console-plugin-cert podName:2ea9caca-0266-4894-ad2e-c02b4e5eeddf nodeName:}" failed. No retries permitted until 2026-07-17 12:26:58.918424288 +0000 UTC m=+263.049808928 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-qn4xk" (UID: "2ea9caca-0266-4894-ad2e-c02b4e5eeddf") : secret "networking-console-plugin-cert" not found
Jul 17 12:26:46.643783 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:46.643747    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:46.643783 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:46.643781    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:46.644237 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:46.644217    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:46.646034 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:46.646018    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97f2ceb2-b1ef-4380-b3ce-8e9ff2498282-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8qwp2\" (UID: \"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282\") " pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:46.712492 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:46.712470    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nmxgd\""
Jul 17 12:26:46.720712 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:46.720692    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8qwp2"
Jul 17 12:26:46.843566 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:46.843532    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8qwp2"]
Jul 17 12:26:46.847617 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:26:46.847594    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f2ceb2_b1ef_4380_b3ce_8e9ff2498282.slice/crio-8eeefbae3811c7ac123d4e9b83891f837de6245aa36bc75fcfffca79f4e039ba WatchSource:0}: Error finding container 8eeefbae3811c7ac123d4e9b83891f837de6245aa36bc75fcfffca79f4e039ba: Status 404 returned error can't find the container with id 8eeefbae3811c7ac123d4e9b83891f837de6245aa36bc75fcfffca79f4e039ba
Jul 17 12:26:47.232120 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:47.232084    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8qwp2" event={"ID":"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282","Type":"ContainerStarted","Data":"8eeefbae3811c7ac123d4e9b83891f837de6245aa36bc75fcfffca79f4e039ba"}
Jul 17 12:26:47.349687 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:47.349142    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs\") pod \"network-metrics-daemon-ml8cr\" (UID: \"e5f413a7-be80-43f3-950e-be9b5c4bf9b4\") " pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:26:47.352586 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:47.352532    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f413a7-be80-43f3-950e-be9b5c4bf9b4-metrics-certs\") pod \"network-metrics-daemon-ml8cr\" (UID: \"e5f413a7-be80-43f3-950e-be9b5c4bf9b4\") " pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:26:47.400505 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:47.400471    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vpz7h\""
Jul 17 12:26:47.408102 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:47.407847    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ml8cr"
Jul 17 12:26:47.570393 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:47.570361    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ml8cr"]
Jul 17 12:26:47.573019 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:26:47.572972    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f413a7_be80_43f3_950e_be9b5c4bf9b4.slice/crio-be207ae88140f8b8e72d74d6ee65813340bf1433bd368493a39726f919024125 WatchSource:0}: Error finding container be207ae88140f8b8e72d74d6ee65813340bf1433bd368493a39726f919024125: Status 404 returned error can't find the container with id be207ae88140f8b8e72d74d6ee65813340bf1433bd368493a39726f919024125
Jul 17 12:26:48.237255 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:48.237216    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ml8cr" event={"ID":"e5f413a7-be80-43f3-950e-be9b5c4bf9b4","Type":"ContainerStarted","Data":"be207ae88140f8b8e72d74d6ee65813340bf1433bd368493a39726f919024125"}
Jul 17 12:26:49.243754 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:49.243678    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ml8cr" event={"ID":"e5f413a7-be80-43f3-950e-be9b5c4bf9b4","Type":"ContainerStarted","Data":"d4cb14e998e7c7bec9db061a90d20f3008266399fa5b3f2b5c114983c8d6a566"}
Jul 17 12:26:49.243754 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:49.243742    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ml8cr" event={"ID":"e5f413a7-be80-43f3-950e-be9b5c4bf9b4","Type":"ContainerStarted","Data":"58a82b819c1c60b040d812d0bcf3f0fb6cfb56128720483099d5e27a96d4f87a"}
Jul 17 12:26:49.260423 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:49.260365    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ml8cr" podStartSLOduration=250.954463231 podStartE2EDuration="4m12.260342057s" totalImagesPullingTime="1.305878826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:26:49.259943443 +0000 UTC m=+253.391328128" watchObservedRunningTime="2026-07-17 12:26:49.260342057 +0000 UTC m=+253.391726720"
Jul 17 12:26:52.256123 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:52.256073    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8qwp2" event={"ID":"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282","Type":"ContainerStarted","Data":"2daa1a18be4b2e08d32b37474afae10dc843a26e66f248786aa37d9ef7d0fede"}
Jul 17 12:26:52.397139 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:52.397068    2565 scope.go:117] "RemoveContainer" containerID="c0923cdb87ab76413a5e62a20e7370d1770e83224f135350c44c737b947aac7f"
Jul 17 12:26:53.261845 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:53.261813    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 12:26:53.262257 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:53.261876    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-p8vsw" event={"ID":"3b0956d7-a5e7-4710-8063-dbc0e5429f9f","Type":"ContainerStarted","Data":"3d1ddc793a61a03a6c1d63f5f655667d46e65c1f72cc3e5441d7a9558f454158"}
Jul 17 12:26:53.262314 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:53.262259    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-p8vsw"
Jul 17 12:26:53.281365 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:53.281307    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-p8vsw" podStartSLOduration=23.811126553 podStartE2EDuration="26.281285137s" totalImagesPullingTime="2.470158584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:26:53.27978507 +0000 UTC m=+257.411169758" watchObservedRunningTime="2026-07-17 12:26:53.281285137 +0000 UTC m=+257.412669799"
Jul 17 12:26:53.364964 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:53.364936    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-p8vsw"
Jul 17 12:26:55.270487 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:55.270462    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8qwp2" event={"ID":"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282","Type":"ContainerStarted","Data":"5dda06d27f83f55cb82517023ec0645cd8a5540ad5b627d11ff2a4d225309fd1"}
Jul 17 12:26:58.950268 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:58.950159    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-qn4xk\" (UID: \"2ea9caca-0266-4894-ad2e-c02b4e5eeddf\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-qn4xk"
Jul 17 12:26:58.952639 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:58.952607    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2ea9caca-0266-4894-ad2e-c02b4e5eeddf-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-qn4xk\" (UID: \"2ea9caca-0266-4894-ad2e-c02b4e5eeddf\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-qn4xk"
Jul 17 12:26:59.034502 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:59.034469    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:26:59.169679 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:59.169654    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-698j8\""
Jul 17 12:26:59.177725 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:59.177708    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-qn4xk"
Jul 17 12:26:59.285172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:59.285142    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8qwp2" event={"ID":"97f2ceb2-b1ef-4380-b3ce-8e9ff2498282","Type":"ContainerStarted","Data":"5ff196410a907131b18e98c3a0436364e0cfda250d98a0470b3e9c1dc3174f89"}
Jul 17 12:26:59.299753 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:59.299702    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-qn4xk"]
Jul 17 12:26:59.302952 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:26:59.302921    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea9caca_0266_4894_ad2e_c02b4e5eeddf.slice/crio-d8bf7e97ffc9b9689b63ea9445b336e7b10dbcfcdc0dd4495e9df957142a3616 WatchSource:0}: Error finding container d8bf7e97ffc9b9689b63ea9445b336e7b10dbcfcdc0dd4495e9df957142a3616: Status 404 returned error can't find the container with id d8bf7e97ffc9b9689b63ea9445b336e7b10dbcfcdc0dd4495e9df957142a3616
Jul 17 12:26:59.303714 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:26:59.303676    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8qwp2" podStartSLOduration=17.698273859 podStartE2EDuration="29.303661846s" totalImagesPullingTime="11.605387987s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 12:26:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:26:59.302973822 +0000 UTC m=+263.434358510" watchObservedRunningTime="2026-07-17 12:26:59.303661846 +0000 UTC m=+263.435046506"
Jul 17 12:27:00.289297 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:00.289260    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-qn4xk" event={"ID":"2ea9caca-0266-4894-ad2e-c02b4e5eeddf","Type":"ContainerStarted","Data":"d8bf7e97ffc9b9689b63ea9445b336e7b10dbcfcdc0dd4495e9df957142a3616"}
Jul 17 12:27:01.089742 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:01.089709    2565 patch_prober.go:28] interesting pod/image-registry-67b89bcb4d-gsdzm container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 17 12:27:01.089905 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:01.089757    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm" podUID="f080f086-e76c-4839-baa2-e07d90fc2a80" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:27:01.293535 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:01.293501    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-qn4xk" event={"ID":"2ea9caca-0266-4894-ad2e-c02b4e5eeddf","Type":"ContainerStarted","Data":"a4c9ddb4740d5d15e1330bce49120efd51e8ac99321493d88f5df0942f3967d0"}
Jul 17 12:27:01.312571 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:01.312531    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-qn4xk" podStartSLOduration=33.301829829 podStartE2EDuration="34.312517671s" totalImagesPullingTime="1.010687842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:27:01.311956557 +0000 UTC m=+265.443341245" watchObservedRunningTime="2026-07-17 12:27:01.312517671 +0000 UTC m=+265.443902331"
Jul 17 12:27:01.461429 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:01.461357    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7vkc7"]
Jul 17 12:27:01.463285 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:01.463271    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7vkc7"
Jul 17 12:27:01.465954 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:01.465937    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-p8w9x\""
Jul 17 12:27:01.466029 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:01.465966    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 17 12:27:01.471777 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:01.471756    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7vkc7"]
Jul 17 12:27:01.570683 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:01.570655    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9b65fed5-8ade-412d-857f-0206503fd06a-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-7vkc7\" (UID: \"9b65fed5-8ade-412d-857f-0206503fd06a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7vkc7"
Jul 17 12:27:01.671702 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:01.671667    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9b65fed5-8ade-412d-857f-0206503fd06a-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-7vkc7\" (UID: \"9b65fed5-8ade-412d-857f-0206503fd06a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7vkc7"
Jul 17 12:27:01.674223 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:01.674202    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9b65fed5-8ade-412d-857f-0206503fd06a-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-7vkc7\" (UID: \"9b65fed5-8ade-412d-857f-0206503fd06a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7vkc7"
Jul 17 12:27:01.772138 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:01.772071    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7vkc7"
Jul 17 12:27:01.892258 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:01.892236    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7vkc7"]
Jul 17 12:27:01.894676 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:27:01.894652    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b65fed5_8ade_412d_857f_0206503fd06a.slice/crio-42a30995fa7a5f85e6b7607975a68530b3f37b8f07e34f81f2aa0242970311f3 WatchSource:0}: Error finding container 42a30995fa7a5f85e6b7607975a68530b3f37b8f07e34f81f2aa0242970311f3: Status 404 returned error can't find the container with id 42a30995fa7a5f85e6b7607975a68530b3f37b8f07e34f81f2aa0242970311f3
Jul 17 12:27:02.298450 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:02.298416    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7vkc7" event={"ID":"9b65fed5-8ade-412d-857f-0206503fd06a","Type":"ContainerStarted","Data":"42a30995fa7a5f85e6b7607975a68530b3f37b8f07e34f81f2aa0242970311f3"}
Jul 17 12:27:03.225520 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:03.225444    2565 patch_prober.go:28] interesting pod/image-registry-67b89bcb4d-gsdzm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 17 12:27:03.225520 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:03.225492    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm" podUID="f080f086-e76c-4839-baa2-e07d90fc2a80" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:27:03.305029 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:03.304995    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7vkc7" event={"ID":"9b65fed5-8ade-412d-857f-0206503fd06a","Type":"ContainerStarted","Data":"022ef0db2e6aee2511392519028bc3bcaf39f49d72c3bec52e11b2df5dde5462"}
Jul 17 12:27:03.305412 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:03.305148    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7vkc7"
Jul 17 12:27:03.309848 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:03.309823    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7vkc7"
Jul 17 12:27:03.322127 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:03.322087    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-7vkc7" podStartSLOduration=1.4034979650000001 podStartE2EDuration="2.32207367s" totalImagesPullingTime="918.575705ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:27:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:27:03.321057832 +0000 UTC m=+267.452442510" watchObservedRunningTime="2026-07-17 12:27:03.32207367 +0000 UTC m=+267.453458328"
Jul 17 12:27:11.090171 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:11.090140    2565 patch_prober.go:28] interesting pod/image-registry-67b89bcb4d-gsdzm container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 17 12:27:11.090516 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:11.090187    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm" podUID="f080f086-e76c-4839-baa2-e07d90fc2a80" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:27:13.224758 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:13.224729    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:27:14.434466 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:14.434431    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67b89bcb4d-gsdzm"]
Jul 17 12:27:15.877239 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.877212    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"]
Jul 17 12:27:15.909481 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.909457    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4k2wr"]
Jul 17 12:27:15.909618 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.909580    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"
Jul 17 12:27:15.911423 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.911404    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"]
Jul 17 12:27:15.911539 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.911516    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:15.912204 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.912189    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-j5rnn\""
Jul 17 12:27:15.912257 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.912245    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 17 12:27:15.913312 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.913298    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 17 12:27:15.913582 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.913565    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 17 12:27:15.913651 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.913588    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 17 12:27:15.913703 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.913647    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 17 12:27:15.913703 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.913613    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 17 12:27:15.913703 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.913588    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 17 12:27:15.913842 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.913781    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 17 12:27:15.913935 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.913920    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rsxtw\""
Jul 17 12:27:15.914074 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.914058    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 17 12:27:15.957220 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.957195    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/14b4f94d-7ff0-4416-aa79-c2ebb50a19ec-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-dqsfh\" (UID: \"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"
Jul 17 12:27:15.957318 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.957224    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-node-exporter-accelerators-collector-config\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:15.957318 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.957243    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14b4f94d-7ff0-4416-aa79-c2ebb50a19ec-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-dqsfh\" (UID: \"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"
Jul 17 12:27:15.957318 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.957266    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-node-exporter-textfile\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:15.957318 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.957292    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcl44\" (UniqueName: \"kubernetes.io/projected/14b4f94d-7ff0-4416-aa79-c2ebb50a19ec-kube-api-access-pcl44\") pod \"kube-state-metrics-56c5594674-dqsfh\" (UID: \"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"
Jul 17 12:27:15.957516 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.957328    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/14b4f94d-7ff0-4416-aa79-c2ebb50a19ec-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-dqsfh\" (UID: \"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"
Jul 17 12:27:15.957516 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.957345    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-sys\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:15.957516 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.957398    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/14b4f94d-7ff0-4416-aa79-c2ebb50a19ec-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-dqsfh\" (UID: \"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"
Jul 17 12:27:15.957516 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.957440    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-metrics-client-ca\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:15.957516 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.957482    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzn84\" (UniqueName: \"kubernetes.io/projected/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-kube-api-access-gzn84\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:15.957700 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.957551    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:15.957700 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.957591    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-node-exporter-tls\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:15.957700 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.957613    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-node-exporter-wtmp\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:15.957700 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.957629    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14b4f94d-7ff0-4416-aa79-c2ebb50a19ec-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-dqsfh\" (UID: \"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"
Jul 17 12:27:15.957700 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:15.957659    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-root\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:16.058617 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.058588    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/14b4f94d-7ff0-4416-aa79-c2ebb50a19ec-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-dqsfh\" (UID: \"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"
Jul 17 12:27:16.058729 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.058626    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-sys\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:16.058729 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.058657    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/14b4f94d-7ff0-4416-aa79-c2ebb50a19ec-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-dqsfh\" (UID: \"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"
Jul 17 12:27:16.058729 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.058686    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-metrics-client-ca\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:16.058729 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.058703    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-sys\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:16.058729 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.058720    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzn84\" (UniqueName: \"kubernetes.io/projected/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-kube-api-access-gzn84\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:16.058964 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.058750    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:16.058964 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.058928    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-node-exporter-tls\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:16.059094 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.058963    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-node-exporter-wtmp\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:16.059094 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.059015    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14b4f94d-7ff0-4416-aa79-c2ebb50a19ec-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-dqsfh\" (UID: \"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"
Jul 17 12:27:16.059094 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:27:16.059045    2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 17 12:27:16.059291 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:27:16.059111    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-node-exporter-tls podName:feebc2f4-3dab-4a0e-93e1-7a39f18ea223 nodeName:}" failed. No retries permitted until 2026-07-17 12:27:16.559092019 +0000 UTC m=+280.690476674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-node-exporter-tls") pod "node-exporter-4k2wr" (UID: "feebc2f4-3dab-4a0e-93e1-7a39f18ea223") : secret "node-exporter-tls" not found
Jul 17 12:27:16.059291 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.059149    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/14b4f94d-7ff0-4416-aa79-c2ebb50a19ec-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-dqsfh\" (UID: \"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"
Jul 17 12:27:16.059291 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.059170    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-node-exporter-wtmp\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:16.059291 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.059047    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-root\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:16.059291 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.059209    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-root\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:16.059291 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.059216    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/14b4f94d-7ff0-4416-aa79-c2ebb50a19ec-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-dqsfh\" (UID: \"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"
Jul 17 12:27:16.059291 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.059249    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-node-exporter-accelerators-collector-config\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:16.059791 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.059299    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-metrics-client-ca\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:16.059791 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.059360    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/14b4f94d-7ff0-4416-aa79-c2ebb50a19ec-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-dqsfh\" (UID: \"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"
Jul 17 12:27:16.059791 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:27:16.059381    2565 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 17 12:27:16.059791 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.059405    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14b4f94d-7ff0-4416-aa79-c2ebb50a19ec-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-dqsfh\" (UID: \"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"
Jul 17 12:27:16.059791 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:27:16.059428    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14b4f94d-7ff0-4416-aa79-c2ebb50a19ec-kube-state-metrics-tls podName:14b4f94d-7ff0-4416-aa79-c2ebb50a19ec nodeName:}" failed. No retries permitted until 2026-07-17 12:27:16.559412276 +0000 UTC m=+280.690796915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/14b4f94d-7ff0-4416-aa79-c2ebb50a19ec-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-dqsfh" (UID: "14b4f94d-7ff0-4416-aa79-c2ebb50a19ec") : secret "kube-state-metrics-tls" not found
Jul 17 12:27:16.059791 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.059461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-node-exporter-textfile\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:16.059791 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.059491    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcl44\" (UniqueName: \"kubernetes.io/projected/14b4f94d-7ff0-4416-aa79-c2ebb50a19ec-kube-api-access-pcl44\") pod \"kube-state-metrics-56c5594674-dqsfh\" (UID: \"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"
Jul 17 12:27:16.059791 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.059681    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-node-exporter-textfile\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:16.060202 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.059907    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-node-exporter-accelerators-collector-config\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:16.060202 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.059927    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14b4f94d-7ff0-4416-aa79-c2ebb50a19ec-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-dqsfh\" (UID: \"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"
Jul 17 12:27:16.061448 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.061423    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:16.061530 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.061503    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14b4f94d-7ff0-4416-aa79-c2ebb50a19ec-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-dqsfh\" (UID: \"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"
Jul 17 12:27:16.068786 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.068766    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzn84\" (UniqueName: \"kubernetes.io/projected/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-kube-api-access-gzn84\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:16.069009 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.068964    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcl44\" (UniqueName: \"kubernetes.io/projected/14b4f94d-7ff0-4416-aa79-c2ebb50a19ec-kube-api-access-pcl44\") pod \"kube-state-metrics-56c5594674-dqsfh\" (UID: \"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"
Jul 17 12:27:16.563374 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.563340    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-node-exporter-tls\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:16.563521 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.563383    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/14b4f94d-7ff0-4416-aa79-c2ebb50a19ec-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-dqsfh\" (UID: \"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"
Jul 17 12:27:16.565768 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.565739    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/feebc2f4-3dab-4a0e-93e1-7a39f18ea223-node-exporter-tls\") pod \"node-exporter-4k2wr\" (UID: \"feebc2f4-3dab-4a0e-93e1-7a39f18ea223\") " pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:16.565877 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.565776    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/14b4f94d-7ff0-4416-aa79-c2ebb50a19ec-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-dqsfh\" (UID: \"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"
Jul 17 12:27:16.820218 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.820147    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"
Jul 17 12:27:16.824609 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.824586    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4k2wr"
Jul 17 12:27:16.833653 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:27:16.833628    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeebc2f4_3dab_4a0e_93e1_7a39f18ea223.slice/crio-47d041fd6e7d3bce5ea86c4bb44bd805eb6268a4aae57418d8035d0433b0940a WatchSource:0}: Error finding container 47d041fd6e7d3bce5ea86c4bb44bd805eb6268a4aae57418d8035d0433b0940a: Status 404 returned error can't find the container with id 47d041fd6e7d3bce5ea86c4bb44bd805eb6268a4aae57418d8035d0433b0940a
Jul 17 12:27:16.938367 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.938339    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 17 12:27:16.941661 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.941643    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:16.944214 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.944190    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 17 12:27:16.944317 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.944190    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 17 12:27:16.944388 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.944193    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 17 12:27:16.944388 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.944196    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 17 12:27:16.944504 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.944455    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 17 12:27:16.944644 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.944626    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 17 12:27:16.944746 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.944630    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-45wfd\""
Jul 17 12:27:16.944746 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.944690    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 17 12:27:16.944913 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.944883    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 17 12:27:16.944913 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.944888    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 17 12:27:16.953800 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.953743    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-dqsfh"]
Jul 17 12:27:16.956800 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:27:16.956737    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b4f94d_7ff0_4416_aa79_c2ebb50a19ec.slice/crio-42c02a23ab6e54eededa5c5eae0e5eabead9e1236a8ed5476f16fb3526df973f WatchSource:0}: Error finding container 42c02a23ab6e54eededa5c5eae0e5eabead9e1236a8ed5476f16fb3526df973f: Status 404 returned error can't find the container with id 42c02a23ab6e54eededa5c5eae0e5eabead9e1236a8ed5476f16fb3526df973f
Jul 17 12:27:16.957548 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:16.957531    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 17 12:27:17.067762 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.067728    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.067866 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.067773    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.067866 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.067801    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.067866 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.067822    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-config-out\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.067866 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.067856    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.068060 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.067911    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-config-volume\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.068060 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.067936    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.068060 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.067975    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.068184 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.068062    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.068184 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.068104    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-web-config\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.068184 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.068140    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.068184 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.068172    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.068324 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.068204    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hds9c\" (UniqueName: \"kubernetes.io/projected/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-kube-api-access-hds9c\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.169025 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.168933    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.169025 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.168978    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.169203 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.169104    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-web-config\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.169203 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.169131    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.169203 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.169153    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.169353 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:27:17.169272    2565 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 17 12:27:17.169353 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.169302    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hds9c\" (UniqueName: \"kubernetes.io/projected/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-kube-api-access-hds9c\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.169353 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.169327    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.169353 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:27:17.169344    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-secret-alertmanager-main-tls podName:6a1de2bb-19b0-4eb1-9b61-16010ec58da3 nodeName:}" failed. No retries permitted until 2026-07-17 12:27:17.66932439 +0000 UTC m=+281.800709060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "6a1de2bb-19b0-4eb1-9b61-16010ec58da3") : secret "alertmanager-main-tls" not found
Jul 17 12:27:17.169530 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.169383    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.169530 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.169413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.169530 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.169438    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.169671 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:27:17.169556    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-alertmanager-trusted-ca-bundle podName:6a1de2bb-19b0-4eb1-9b61-16010ec58da3 nodeName:}" failed. No retries permitted until 2026-07-17 12:27:17.669535811 +0000 UTC m=+281.800920454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "6a1de2bb-19b0-4eb1-9b61-16010ec58da3") : configmap references non-existent config key: ca-bundle.crt
Jul 17 12:27:17.169671 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.169580    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-config-out\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.169671 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.169630    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.169671 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.169668    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-config-volume\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.169858 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.169695    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.170595 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.170565    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.172388 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.172329    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.172388 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.172337    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-config-out\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.172525 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.172424    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.172591 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.172566    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-web-config\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.172711 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.172689    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.172768 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.172689    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.172855 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.172839    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.173065 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.173047    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-config-volume\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.179737 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.179719    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hds9c\" (UniqueName: \"kubernetes.io/projected/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-kube-api-access-hds9c\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.349978 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.349942    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh" event={"ID":"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec","Type":"ContainerStarted","Data":"42c02a23ab6e54eededa5c5eae0e5eabead9e1236a8ed5476f16fb3526df973f"}
Jul 17 12:27:17.351248 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.351220    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4k2wr" event={"ID":"feebc2f4-3dab-4a0e-93e1-7a39f18ea223","Type":"ContainerStarted","Data":"47d041fd6e7d3bce5ea86c4bb44bd805eb6268a4aae57418d8035d0433b0940a"}
Jul 17 12:27:17.675633 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.675591    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.675824 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.675678    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.676655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.676628    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.678728 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.678701    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6a1de2bb-19b0-4eb1-9b61-16010ec58da3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6a1de2bb-19b0-4eb1-9b61-16010ec58da3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:17.851047 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:17.851023    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 17 12:27:18.271562 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.271531    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 17 12:27:18.272440 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:27:18.272402    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1de2bb_19b0_4eb1_9b61_16010ec58da3.slice/crio-eec4de3b2322b170225ba8dfa4c6586c295264647746681a8bd5848ad4b6e866 WatchSource:0}: Error finding container eec4de3b2322b170225ba8dfa4c6586c295264647746681a8bd5848ad4b6e866: Status 404 returned error can't find the container with id eec4de3b2322b170225ba8dfa4c6586c295264647746681a8bd5848ad4b6e866
Jul 17 12:27:18.359813 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.359779    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh" event={"ID":"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec","Type":"ContainerStarted","Data":"a0d2268db2b94271eae96dd8dcf778741656eb08f5620738b96219f54c14262b"}
Jul 17 12:27:18.359899 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.359842    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh" event={"ID":"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec","Type":"ContainerStarted","Data":"85f8180c2cc3576a6317fa8181a90c3daffb666492054ceb1d13968525d29755"}
Jul 17 12:27:18.361038 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.361012    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a1de2bb-19b0-4eb1-9b61-16010ec58da3","Type":"ContainerStarted","Data":"eec4de3b2322b170225ba8dfa4c6586c295264647746681a8bd5848ad4b6e866"}
Jul 17 12:27:18.362506 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.362481    2565 generic.go:356] "Generic (PLEG): container finished" podID="feebc2f4-3dab-4a0e-93e1-7a39f18ea223" containerID="bfcf4e7beaf5f1022e07d145a8fd8af4547e0c39a8d246c0763fb774d797787b" exitCode=0
Jul 17 12:27:18.362610 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.362565    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4k2wr" event={"ID":"feebc2f4-3dab-4a0e-93e1-7a39f18ea223","Type":"ContainerDied","Data":"bfcf4e7beaf5f1022e07d145a8fd8af4547e0c39a8d246c0763fb774d797787b"}
Jul 17 12:27:18.855419 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.855386    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-75df9fcb9d-lct87"]
Jul 17 12:27:18.858399 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.858377    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:18.861516 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.861182    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6h4c6\""
Jul 17 12:27:18.861516 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.861194    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 17 12:27:18.861516 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.861340    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 17 12:27:18.861516 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.861377    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 17 12:27:18.861790 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.861723    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 17 12:27:18.861790 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.861761    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9omlufmjtfuto\""
Jul 17 12:27:18.862253 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.862235    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 17 12:27:18.873657 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.873634    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75df9fcb9d-lct87"]
Jul 17 12:27:18.993294 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.993268    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v74cv\" (UniqueName: \"kubernetes.io/projected/4b898525-cf0d-473b-8b77-d477947facf0-kube-api-access-v74cv\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:18.993451 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.993305    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b898525-cf0d-473b-8b77-d477947facf0-metrics-client-ca\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:18.993451 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.993328    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4b898525-cf0d-473b-8b77-d477947facf0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:18.993451 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.993381    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4b898525-cf0d-473b-8b77-d477947facf0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:18.993451 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.993419    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4b898525-cf0d-473b-8b77-d477947facf0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:18.993594 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.993485    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4b898525-cf0d-473b-8b77-d477947facf0-secret-thanos-querier-tls\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:18.993594 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.993514    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4b898525-cf0d-473b-8b77-d477947facf0-secret-grpc-tls\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:18.993594 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:18.993546    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4b898525-cf0d-473b-8b77-d477947facf0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:19.000414 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:27:19.000389    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-gbnmx" podUID="793d61b2-520e-416a-b856-545c0a0f45b5"
Jul 17 12:27:19.094784 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.094751    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4b898525-cf0d-473b-8b77-d477947facf0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:19.095050 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.094795    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4b898525-cf0d-473b-8b77-d477947facf0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:19.095050 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.094818    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4b898525-cf0d-473b-8b77-d477947facf0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:19.095050 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.094851    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4b898525-cf0d-473b-8b77-d477947facf0-secret-thanos-querier-tls\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:19.095050 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.094871    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4b898525-cf0d-473b-8b77-d477947facf0-secret-grpc-tls\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:19.095050 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.094901    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4b898525-cf0d-473b-8b77-d477947facf0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:19.095469 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.095440    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v74cv\" (UniqueName: \"kubernetes.io/projected/4b898525-cf0d-473b-8b77-d477947facf0-kube-api-access-v74cv\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:19.095699 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.095507    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b898525-cf0d-473b-8b77-d477947facf0-metrics-client-ca\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:19.096532 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.096503    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b898525-cf0d-473b-8b77-d477947facf0-metrics-client-ca\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:19.097709 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.097645    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4b898525-cf0d-473b-8b77-d477947facf0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:19.097876 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.097838    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4b898525-cf0d-473b-8b77-d477947facf0-secret-thanos-querier-tls\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:19.098062 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.098040    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4b898525-cf0d-473b-8b77-d477947facf0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:19.098199 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.098180    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4b898525-cf0d-473b-8b77-d477947facf0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:19.098535 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.098516    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4b898525-cf0d-473b-8b77-d477947facf0-secret-grpc-tls\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:19.098726 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.098708    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4b898525-cf0d-473b-8b77-d477947facf0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:19.105286 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.105263    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v74cv\" (UniqueName: \"kubernetes.io/projected/4b898525-cf0d-473b-8b77-d477947facf0-kube-api-access-v74cv\") pod \"thanos-querier-75df9fcb9d-lct87\" (UID: \"4b898525-cf0d-473b-8b77-d477947facf0\") " pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:19.171688 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.171625    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:19.373818 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.373728    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh" event={"ID":"14b4f94d-7ff0-4416-aa79-c2ebb50a19ec","Type":"ContainerStarted","Data":"32194a48abce2687a2c18cfca823c982a855dcc5c02dd3f9af7d0e853cd66aa2"}
Jul 17 12:27:19.377796 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.377511    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:27:19.377796 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.377550    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4k2wr" event={"ID":"feebc2f4-3dab-4a0e-93e1-7a39f18ea223","Type":"ContainerStarted","Data":"4657441387fe45d191015605da436f43a2f877bdd54c6016a24a778b1f4af26b"}
Jul 17 12:27:19.377796 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.377607    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4k2wr" event={"ID":"feebc2f4-3dab-4a0e-93e1-7a39f18ea223","Type":"ContainerStarted","Data":"165c7933656aeea0b23b3a19ee86583a48d85fe914551acad0d6775fb2a17612"}
Jul 17 12:27:19.394999 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.394927    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-dqsfh" podStartSLOduration=3.195416534 podStartE2EDuration="4.394908539s" totalImagesPullingTime="1.199492005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:27:19.392847131 +0000 UTC m=+283.524231832" watchObservedRunningTime="2026-07-17 12:27:19.394908539 +0000 UTC m=+283.526293381"
Jul 17 12:27:19.413650 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.413595    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4k2wr" podStartSLOduration=3.404549762 podStartE2EDuration="4.41357383s" totalImagesPullingTime="931.270572ms" totalInitContainerRuntime="77.753496ms" isStatefulPod=true podCreationTimestamp="2026-07-17 12:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:27:19.412641596 +0000 UTC m=+283.544026283" watchObservedRunningTime="2026-07-17 12:27:19.41357383 +0000 UTC m=+283.544958493"
Jul 17 12:27:19.448212 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:19.448179    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-75df9fcb9d-lct87"]
Jul 17 12:27:19.506804 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:27:19.506767    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b898525_cf0d_473b_8b77_d477947facf0.slice/crio-8e92b52cc41918c3220ef074233396328ad8b7eefcfafa0899873ef601500de6 WatchSource:0}: Error finding container 8e92b52cc41918c3220ef074233396328ad8b7eefcfafa0899873ef601500de6: Status 404 returned error can't find the container with id 8e92b52cc41918c3220ef074233396328ad8b7eefcfafa0899873ef601500de6
Jul 17 12:27:20.383535 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:20.383486    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87" event={"ID":"4b898525-cf0d-473b-8b77-d477947facf0","Type":"ContainerStarted","Data":"8e92b52cc41918c3220ef074233396328ad8b7eefcfafa0899873ef601500de6"}
Jul 17 12:27:20.385604 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:20.385577    2565 generic.go:356] "Generic (PLEG): container finished" podID="6a1de2bb-19b0-4eb1-9b61-16010ec58da3" containerID="38641e78b98c16030bf64c4811859cec676a92320c4181475c8056c3728fe584" exitCode=0
Jul 17 12:27:20.385732 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:20.385661    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a1de2bb-19b0-4eb1-9b61-16010ec58da3","Type":"ContainerDied","Data":"38641e78b98c16030bf64c4811859cec676a92320c4181475c8056c3728fe584"}
Jul 17 12:27:22.323724 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:22.323689    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls\") pod \"dns-default-gbnmx\" (UID: \"793d61b2-520e-416a-b856-545c0a0f45b5\") " pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:27:22.326009 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:22.325976    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/793d61b2-520e-416a-b856-545c0a0f45b5-metrics-tls\") pod \"dns-default-gbnmx\" (UID: \"793d61b2-520e-416a-b856-545c0a0f45b5\") " pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:27:22.381086 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:22.381033    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gk42k\""
Jul 17 12:27:22.389420 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:22.389407    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:27:22.406000 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:22.403392    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a1de2bb-19b0-4eb1-9b61-16010ec58da3","Type":"ContainerStarted","Data":"320fa96a87683ba6bd20d07b53bf9660c9d432ca63e527a09fb1cc5edb205f37"}
Jul 17 12:27:22.406000 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:22.403467    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a1de2bb-19b0-4eb1-9b61-16010ec58da3","Type":"ContainerStarted","Data":"ae67ade4835eb7b61ac7dc5f62d3deae177cd293b6028d048df6c879428cb1af"}
Jul 17 12:27:22.406000 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:22.403504    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a1de2bb-19b0-4eb1-9b61-16010ec58da3","Type":"ContainerStarted","Data":"1752b5a7c47b4da94393a45c6ec4245eb44fada3e2848098a846eeea063f98f5"}
Jul 17 12:27:22.406000 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:22.403547    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a1de2bb-19b0-4eb1-9b61-16010ec58da3","Type":"ContainerStarted","Data":"cf6f2ee4748f0625fc151ffdd59cb832bdf329add25f014bdcee72f35ac2b870"}
Jul 17 12:27:22.406000 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:22.403583    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a1de2bb-19b0-4eb1-9b61-16010ec58da3","Type":"ContainerStarted","Data":"9a0e0623611e18e80c3dfea74a1aff18a22aff8e406f6e65f72683b7fbe98029"}
Jul 17 12:27:22.406000 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:22.404759    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87" event={"ID":"4b898525-cf0d-473b-8b77-d477947facf0","Type":"ContainerStarted","Data":"e47ef6407618d927fb7191e66ed3f4d14f56ec31a7f3c3978f06dd79615fca52"}
Jul 17 12:27:22.406000 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:22.404823    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87" event={"ID":"4b898525-cf0d-473b-8b77-d477947facf0","Type":"ContainerStarted","Data":"48e1e37aa7d98e2987480fa3ec8b11ba720086ea1befa0a559008cf599f39103"}
Jul 17 12:27:22.406000 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:22.404861    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87" event={"ID":"4b898525-cf0d-473b-8b77-d477947facf0","Type":"ContainerStarted","Data":"4e0a76b1740852d9aa10ab7cf7bea019813a6045bc89180851ed989e691d20ab"}
Jul 17 12:27:22.529319 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:22.529293    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gbnmx"]
Jul 17 12:27:22.531825 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:27:22.531791    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793d61b2_520e_416a_b856_545c0a0f45b5.slice/crio-0c42969310e3816a4495667585e4dc34b7c9dddb32c128d2883f455f93953a3b WatchSource:0}: Error finding container 0c42969310e3816a4495667585e4dc34b7c9dddb32c128d2883f455f93953a3b: Status 404 returned error can't find the container with id 0c42969310e3816a4495667585e4dc34b7c9dddb32c128d2883f455f93953a3b
Jul 17 12:27:23.413381 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:23.413343    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87" event={"ID":"4b898525-cf0d-473b-8b77-d477947facf0","Type":"ContainerStarted","Data":"af3c0ce74cd369a307a7d56eae0ee6f976fd1ec71e8f90fa517326c1509908f7"}
Jul 17 12:27:23.413820 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:23.413411    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87" event={"ID":"4b898525-cf0d-473b-8b77-d477947facf0","Type":"ContainerStarted","Data":"8a9fcaf499e8a71ac86c623d34df932dab3448d25666ce9dda666319c2a60224"}
Jul 17 12:27:23.413820 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:23.413455    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87" event={"ID":"4b898525-cf0d-473b-8b77-d477947facf0","Type":"ContainerStarted","Data":"c319b10d2fc211bd27f46ebad85660bdc4840c84448a2330e4ca8035c3b02250"}
Jul 17 12:27:23.413820 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:23.413500    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:23.414526 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:23.414498    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gbnmx" event={"ID":"793d61b2-520e-416a-b856-545c0a0f45b5","Type":"ContainerStarted","Data":"0c42969310e3816a4495667585e4dc34b7c9dddb32c128d2883f455f93953a3b"}
Jul 17 12:27:23.417400 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:23.417380    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a1de2bb-19b0-4eb1-9b61-16010ec58da3","Type":"ContainerStarted","Data":"73034179bcec5fd6f99ff467913fed4e196c8bbd7e6464a18a7e4cfd183e6e0e"}
Jul 17 12:27:23.442510 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:23.442466    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87" podStartSLOduration=2.379259771 podStartE2EDuration="5.4424523s" totalImagesPullingTime="3.063192529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:27:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:27:23.440378573 +0000 UTC m=+287.571763233" watchObservedRunningTime="2026-07-17 12:27:23.4424523 +0000 UTC m=+287.573836960"
Jul 17 12:27:23.473188 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:23.473139    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.306999501 podStartE2EDuration="7.473119779s" totalImagesPullingTime="3.146512007s" totalInitContainerRuntime="19.608271ms" isStatefulPod=false podCreationTimestamp="2026-07-17 12:27:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:27:23.470282072 +0000 UTC m=+287.601666743" watchObservedRunningTime="2026-07-17 12:27:23.473119779 +0000 UTC m=+287.604504441"
Jul 17 12:27:24.426920 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:24.426885    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gbnmx" event={"ID":"793d61b2-520e-416a-b856-545c0a0f45b5","Type":"ContainerStarted","Data":"c68b9ffb731e7a8d065ae7753e100c711896fe6691736718abd3ce326b5ce315"}
Jul 17 12:27:24.427328 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:24.426970    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gbnmx" event={"ID":"793d61b2-520e-416a-b856-545c0a0f45b5","Type":"ContainerStarted","Data":"b128b2ef32d11e32d7a257cbc9ccb76c4d54df0df5882994f8ee6c507aaae97b"}
Jul 17 12:27:24.427540 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:24.427523    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:27:24.445100 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:24.445057    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gbnmx" podStartSLOduration=250.943844198 podStartE2EDuration="4m12.445044115s" totalImagesPullingTime="1.501199917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:27:24.444220275 +0000 UTC m=+288.575604963" watchObservedRunningTime="2026-07-17 12:27:24.445044115 +0000 UTC m=+288.576428776"
Jul 17 12:27:29.433883 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:29.433854    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-75df9fcb9d-lct87"
Jul 17 12:27:35.436414 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:35.436387    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gbnmx"
Jul 17 12:27:36.283761 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:36.283730    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 12:27:36.284075 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:36.284061    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 12:27:36.289611 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:36.289590    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 12:27:36.289740 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:36.289710    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 12:27:36.294878 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:36.294863    2565 kubelet.go:1648] "Image garbage collection succeeded"
Jul 17 12:27:39.458623 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.458568    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm" podUID="f080f086-e76c-4839-baa2-e07d90fc2a80" containerName="registry" containerID="cri-o://69c34abe2911d4e75183a1c7223892dac51a7649d422302cac63da8a9a69dc88" gracePeriod=30
Jul 17 12:27:39.693676 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.693653    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:27:39.857940 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.857909    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f080f086-e76c-4839-baa2-e07d90fc2a80-ca-trust-extracted\") pod \"f080f086-e76c-4839-baa2-e07d90fc2a80\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") "
Jul 17 12:27:39.858113 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.857950    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f080f086-e76c-4839-baa2-e07d90fc2a80-trusted-ca\") pod \"f080f086-e76c-4839-baa2-e07d90fc2a80\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") "
Jul 17 12:27:39.858113 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.857976    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f080f086-e76c-4839-baa2-e07d90fc2a80-registry-tls\") pod \"f080f086-e76c-4839-baa2-e07d90fc2a80\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") "
Jul 17 12:27:39.858113 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.858007    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28q6t\" (UniqueName: \"kubernetes.io/projected/f080f086-e76c-4839-baa2-e07d90fc2a80-kube-api-access-28q6t\") pod \"f080f086-e76c-4839-baa2-e07d90fc2a80\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") "
Jul 17 12:27:39.858113 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.858049    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f080f086-e76c-4839-baa2-e07d90fc2a80-bound-sa-token\") pod \"f080f086-e76c-4839-baa2-e07d90fc2a80\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") "
Jul 17 12:27:39.858113 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.858082    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f080f086-e76c-4839-baa2-e07d90fc2a80-registry-certificates\") pod \"f080f086-e76c-4839-baa2-e07d90fc2a80\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") "
Jul 17 12:27:39.858353 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.858116    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f080f086-e76c-4839-baa2-e07d90fc2a80-image-registry-private-configuration\") pod \"f080f086-e76c-4839-baa2-e07d90fc2a80\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") "
Jul 17 12:27:39.858353 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.858145    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f080f086-e76c-4839-baa2-e07d90fc2a80-installation-pull-secrets\") pod \"f080f086-e76c-4839-baa2-e07d90fc2a80\" (UID: \"f080f086-e76c-4839-baa2-e07d90fc2a80\") "
Jul 17 12:27:39.858670 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.858447    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f080f086-e76c-4839-baa2-e07d90fc2a80-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f080f086-e76c-4839-baa2-e07d90fc2a80" (UID: "f080f086-e76c-4839-baa2-e07d90fc2a80"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:27:39.858670 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.858624    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f080f086-e76c-4839-baa2-e07d90fc2a80-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f080f086-e76c-4839-baa2-e07d90fc2a80" (UID: "f080f086-e76c-4839-baa2-e07d90fc2a80"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:27:39.860465 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.860433    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f080f086-e76c-4839-baa2-e07d90fc2a80-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f080f086-e76c-4839-baa2-e07d90fc2a80" (UID: "f080f086-e76c-4839-baa2-e07d90fc2a80"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:27:39.860679 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.860655    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f080f086-e76c-4839-baa2-e07d90fc2a80-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f080f086-e76c-4839-baa2-e07d90fc2a80" (UID: "f080f086-e76c-4839-baa2-e07d90fc2a80"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:27:39.860757 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.860705    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f080f086-e76c-4839-baa2-e07d90fc2a80-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f080f086-e76c-4839-baa2-e07d90fc2a80" (UID: "f080f086-e76c-4839-baa2-e07d90fc2a80"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:27:39.860928 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.860907    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f080f086-e76c-4839-baa2-e07d90fc2a80-kube-api-access-28q6t" (OuterVolumeSpecName: "kube-api-access-28q6t") pod "f080f086-e76c-4839-baa2-e07d90fc2a80" (UID: "f080f086-e76c-4839-baa2-e07d90fc2a80"). InnerVolumeSpecName "kube-api-access-28q6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:27:39.861001 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.860959    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f080f086-e76c-4839-baa2-e07d90fc2a80-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f080f086-e76c-4839-baa2-e07d90fc2a80" (UID: "f080f086-e76c-4839-baa2-e07d90fc2a80"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:27:39.866946 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.866924    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f080f086-e76c-4839-baa2-e07d90fc2a80-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f080f086-e76c-4839-baa2-e07d90fc2a80" (UID: "f080f086-e76c-4839-baa2-e07d90fc2a80"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:27:39.959518 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.959487    2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f080f086-e76c-4839-baa2-e07d90fc2a80-installation-pull-secrets\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:27:39.959518 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.959513    2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f080f086-e76c-4839-baa2-e07d90fc2a80-ca-trust-extracted\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:27:39.959518 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.959524    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f080f086-e76c-4839-baa2-e07d90fc2a80-trusted-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:27:39.959720 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.959532    2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f080f086-e76c-4839-baa2-e07d90fc2a80-registry-tls\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:27:39.959720 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.959540    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28q6t\" (UniqueName: \"kubernetes.io/projected/f080f086-e76c-4839-baa2-e07d90fc2a80-kube-api-access-28q6t\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:27:39.959720 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.959548    2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f080f086-e76c-4839-baa2-e07d90fc2a80-bound-sa-token\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:27:39.959720 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.959556    2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f080f086-e76c-4839-baa2-e07d90fc2a80-registry-certificates\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:27:39.959720 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:39.959566    2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f080f086-e76c-4839-baa2-e07d90fc2a80-image-registry-private-configuration\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:27:40.489428 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:40.489389    2565 generic.go:356] "Generic (PLEG): container finished" podID="f080f086-e76c-4839-baa2-e07d90fc2a80" containerID="69c34abe2911d4e75183a1c7223892dac51a7649d422302cac63da8a9a69dc88" exitCode=0
Jul 17 12:27:40.489901 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:40.489441    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm"
Jul 17 12:27:40.489901 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:40.489479    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm" event={"ID":"f080f086-e76c-4839-baa2-e07d90fc2a80","Type":"ContainerDied","Data":"69c34abe2911d4e75183a1c7223892dac51a7649d422302cac63da8a9a69dc88"}
Jul 17 12:27:40.489901 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:40.489526    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67b89bcb4d-gsdzm" event={"ID":"f080f086-e76c-4839-baa2-e07d90fc2a80","Type":"ContainerDied","Data":"51c18c1c1a045219cfe3bf750bbb401b4a1c90c855bb4eed2ea521088cda3e59"}
Jul 17 12:27:40.489901 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:40.489548    2565 scope.go:117] "RemoveContainer" containerID="69c34abe2911d4e75183a1c7223892dac51a7649d422302cac63da8a9a69dc88"
Jul 17 12:27:40.498047 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:40.498029    2565 scope.go:117] "RemoveContainer" containerID="69c34abe2911d4e75183a1c7223892dac51a7649d422302cac63da8a9a69dc88"
Jul 17 12:27:40.498344 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:27:40.498321    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c34abe2911d4e75183a1c7223892dac51a7649d422302cac63da8a9a69dc88\": container with ID starting with 69c34abe2911d4e75183a1c7223892dac51a7649d422302cac63da8a9a69dc88 not found: ID does not exist" containerID="69c34abe2911d4e75183a1c7223892dac51a7649d422302cac63da8a9a69dc88"
Jul 17 12:27:40.498417 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:40.498357    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c34abe2911d4e75183a1c7223892dac51a7649d422302cac63da8a9a69dc88"} err="failed to get container status \"69c34abe2911d4e75183a1c7223892dac51a7649d422302cac63da8a9a69dc88\": rpc error: code = NotFound desc = could not find container \"69c34abe2911d4e75183a1c7223892dac51a7649d422302cac63da8a9a69dc88\": container with ID starting with 69c34abe2911d4e75183a1c7223892dac51a7649d422302cac63da8a9a69dc88 not found: ID does not exist"
Jul 17 12:27:40.506770 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:40.506749    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67b89bcb4d-gsdzm"]
Jul 17 12:27:40.510082 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:40.510058    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67b89bcb4d-gsdzm"]
Jul 17 12:27:42.401742 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:42.401714    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f080f086-e76c-4839-baa2-e07d90fc2a80" path="/var/lib/kubelet/pods/f080f086-e76c-4839-baa2-e07d90fc2a80/volumes"
Jul 17 12:27:45.509334 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:45.509297    2565 generic.go:356] "Generic (PLEG): container finished" podID="b4b08097-9f28-46c0-95c0-404d419f9cac" containerID="75a953442ed7fa5934475a1fb37fa954c7420e358256a24c8717da6f5023acee" exitCode=0
Jul 17 12:27:45.509711 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:45.509372    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2mbv7" event={"ID":"b4b08097-9f28-46c0-95c0-404d419f9cac","Type":"ContainerDied","Data":"75a953442ed7fa5934475a1fb37fa954c7420e358256a24c8717da6f5023acee"}
Jul 17 12:27:45.509711 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:45.509682    2565 scope.go:117] "RemoveContainer" containerID="75a953442ed7fa5934475a1fb37fa954c7420e358256a24c8717da6f5023acee"
Jul 17 12:27:46.514345 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:46.514311    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2mbv7" event={"ID":"b4b08097-9f28-46c0-95c0-404d419f9cac","Type":"ContainerStarted","Data":"9be109b98495710169d4789633f352e9f8e3ad519bba368f29cb86f69367b117"}
Jul 17 12:27:53.883144 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:27:53.883099    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:28:12.932325 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:28:12.932252    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:28:38.218296 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:28:38.218260    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:28:43.890309 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:28:43.890270    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:30:01.095878 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:30:01.095805    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:30:09.122590 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:30:09.122560    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:30:55.251134 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:30:55.251099    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:31:49.617134 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:31:49.617097    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:32:36.310349 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:32:36.310322    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 12:32:36.310910 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:32:36.310893    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 12:32:36.315074 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:32:36.315054    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 12:32:36.315461 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:32:36.315445    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 12:33:24.275345 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:33:24.275311    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:34:27.223803 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:34:27.223720    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:36:02.356683 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:36:02.356601    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:37:36.338181 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:37:36.338106    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 12:37:36.340072 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:37:36.340053    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 12:37:36.343587 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:37:36.343561    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 12:37:36.345218 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:37:36.345201    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 12:37:36.928097 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:37:36.928052    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:08.951543 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:08.951505    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:09.519113 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:09.519076    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:10.044018 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:10.043975    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:10.571919 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:10.571882    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:11.095441 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:11.095415    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:12.117205 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:12.117174    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:18.335126 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:18.335084    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:18.877043 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:18.877005    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:19.447706 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:19.447682    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:25.459778 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:25.459732    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:25.951739 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:25.951707    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:26.509696 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:26.509659    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:26.984571 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:26.984512    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:36.460878 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:36.460832    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:37.002148 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:37.002114    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:37.559826 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:37.559758    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:38.125891 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:38.125856    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:38.517419 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:38.517335    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:40.504690 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:40.504651    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:41.053825 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:41.053789    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:41.546561 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:41.546454    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:42.056708 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:42.056675    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:53.976674 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:53.976624    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:56.395642 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:56.395600    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:56.924483 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:56.924436    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:57.441844 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:57.441814    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:57.936312 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:57.936279    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:58.798510 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:58.798473    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:38:59.292007 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:38:59.291907    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:05.479893 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:05.479857    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:09.532752 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:09.532708    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:16.468160 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:16.468114    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:20.481575 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:20.481523    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:21.040803 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:21.040762    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:21.543425 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:21.543394    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:22.049287 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:22.049255    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:22.582056 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:22.582027    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:25.006682 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:25.006643    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:25.873610 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:25.873575    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:29.461703 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:29.461663    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:29.975032 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:29.975000    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:30.477113 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:30.477072    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:32.977383 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:32.977344    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:38.542791 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:38.542749    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:39.040510 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:39.040480    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:39.526757 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:39.526723    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:40.017207 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:40.017171    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:41.128557 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:41.128519    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:41.496785 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:41.496681    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:41.978935 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:41.978904    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:42.467388 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:42.467356    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:42.946824 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:42.946788    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:43.529762 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:43.529716    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:44.165040 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:44.165002    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:44.758178 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:44.758139    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:45.302326 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:45.302292    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:45.836875 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:45.836846    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:46.477617 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:46.477587    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:47.036681 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:47.036651    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:39:55.566361 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:39:55.566316    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:11.904638 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:11.904605    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:11.998402 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:11.998372    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:12.534629 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.534574    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78"]
Jul 17 12:40:12.534915 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.534899    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f080f086-e76c-4839-baa2-e07d90fc2a80" containerName="registry"
Jul 17 12:40:12.535025 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.534920    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f080f086-e76c-4839-baa2-e07d90fc2a80" containerName="registry"
Jul 17 12:40:12.535025 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.535003    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f080f086-e76c-4839-baa2-e07d90fc2a80" containerName="registry"
Jul 17 12:40:12.538022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.538006    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78"
Jul 17 12:40:12.540966 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.540949    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 17 12:40:12.541054 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.540974    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xmv68\""
Jul 17 12:40:12.542125 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.542109    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 17 12:40:12.557486 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.557466    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78"]
Jul 17 12:40:12.588904 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.588880    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c0c7926-4319-4e9d-8788-31f7ec35fbba-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78\" (UID: \"1c0c7926-4319-4e9d-8788-31f7ec35fbba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78"
Jul 17 12:40:12.589035 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.588910    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvsfn\" (UniqueName: \"kubernetes.io/projected/1c0c7926-4319-4e9d-8788-31f7ec35fbba-kube-api-access-zvsfn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78\" (UID: \"1c0c7926-4319-4e9d-8788-31f7ec35fbba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78"
Jul 17 12:40:12.589035 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.588938    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c0c7926-4319-4e9d-8788-31f7ec35fbba-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78\" (UID: \"1c0c7926-4319-4e9d-8788-31f7ec35fbba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78"
Jul 17 12:40:12.689827 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.689798    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c0c7926-4319-4e9d-8788-31f7ec35fbba-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78\" (UID: \"1c0c7926-4319-4e9d-8788-31f7ec35fbba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78"
Jul 17 12:40:12.689923 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.689831    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvsfn\" (UniqueName: \"kubernetes.io/projected/1c0c7926-4319-4e9d-8788-31f7ec35fbba-kube-api-access-zvsfn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78\" (UID: \"1c0c7926-4319-4e9d-8788-31f7ec35fbba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78"
Jul 17 12:40:12.689923 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.689865    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c0c7926-4319-4e9d-8788-31f7ec35fbba-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78\" (UID: \"1c0c7926-4319-4e9d-8788-31f7ec35fbba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78"
Jul 17 12:40:12.690257 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.690236    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c0c7926-4319-4e9d-8788-31f7ec35fbba-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78\" (UID: \"1c0c7926-4319-4e9d-8788-31f7ec35fbba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78"
Jul 17 12:40:12.690296 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.690259    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c0c7926-4319-4e9d-8788-31f7ec35fbba-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78\" (UID: \"1c0c7926-4319-4e9d-8788-31f7ec35fbba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78"
Jul 17 12:40:12.699868 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.699847    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvsfn\" (UniqueName: \"kubernetes.io/projected/1c0c7926-4319-4e9d-8788-31f7ec35fbba-kube-api-access-zvsfn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78\" (UID: \"1c0c7926-4319-4e9d-8788-31f7ec35fbba\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78"
Jul 17 12:40:12.796419 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.796353    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:12.846775 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.846753    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78"
Jul 17 12:40:12.892483 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.892453    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:12.976716 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.976695    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78"]
Jul 17 12:40:12.979135 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:40:12.979112    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0c7926_4319_4e9d_8788_31f7ec35fbba.slice/crio-aa1751ce35a06c23ea9802fc736e02fb27cf2b2b895a0ab2d4a61c171220ce94 WatchSource:0}: Error finding container aa1751ce35a06c23ea9802fc736e02fb27cf2b2b895a0ab2d4a61c171220ce94: Status 404 returned error can't find the container with id aa1751ce35a06c23ea9802fc736e02fb27cf2b2b895a0ab2d4a61c171220ce94
Jul 17 12:40:12.980944 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:12.980928    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 12:40:13.234858 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:13.234828    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78" event={"ID":"1c0c7926-4319-4e9d-8788-31f7ec35fbba","Type":"ContainerStarted","Data":"aa1751ce35a06c23ea9802fc736e02fb27cf2b2b895a0ab2d4a61c171220ce94"}
Jul 17 12:40:20.263970 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:20.263900    2565 generic.go:356] "Generic (PLEG): container finished" podID="1c0c7926-4319-4e9d-8788-31f7ec35fbba" containerID="182455d9dd35a269aa8be70e18431e8b82cdfed1940955fad66ee56dacbfc424" exitCode=0
Jul 17 12:40:20.263970 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:20.263960    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78" event={"ID":"1c0c7926-4319-4e9d-8788-31f7ec35fbba","Type":"ContainerDied","Data":"182455d9dd35a269aa8be70e18431e8b82cdfed1940955fad66ee56dacbfc424"}
Jul 17 12:40:23.278520 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:23.278484    2565 generic.go:356] "Generic (PLEG): container finished" podID="1c0c7926-4319-4e9d-8788-31f7ec35fbba" containerID="d54b3bb9fa7f3a26d165629a9c2aae7880f6ab2024100548b0296380b01736d0" exitCode=0
Jul 17 12:40:23.278910 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:23.278562    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78" event={"ID":"1c0c7926-4319-4e9d-8788-31f7ec35fbba","Type":"ContainerDied","Data":"d54b3bb9fa7f3a26d165629a9c2aae7880f6ab2024100548b0296380b01736d0"}
Jul 17 12:40:31.313063 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:31.313023    2565 generic.go:356] "Generic (PLEG): container finished" podID="1c0c7926-4319-4e9d-8788-31f7ec35fbba" containerID="fb9b78a52807794ff0bf00e3eaead7d386eeccf5beebcea4715f2ae3f7718750" exitCode=0
Jul 17 12:40:31.313473 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:31.313080    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78" event={"ID":"1c0c7926-4319-4e9d-8788-31f7ec35fbba","Type":"ContainerDied","Data":"fb9b78a52807794ff0bf00e3eaead7d386eeccf5beebcea4715f2ae3f7718750"}
Jul 17 12:40:32.451486 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:32.451463    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78"
Jul 17 12:40:32.554517 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:32.554480    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c0c7926-4319-4e9d-8788-31f7ec35fbba-util\") pod \"1c0c7926-4319-4e9d-8788-31f7ec35fbba\" (UID: \"1c0c7926-4319-4e9d-8788-31f7ec35fbba\") "
Jul 17 12:40:32.554517 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:32.554533    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvsfn\" (UniqueName: \"kubernetes.io/projected/1c0c7926-4319-4e9d-8788-31f7ec35fbba-kube-api-access-zvsfn\") pod \"1c0c7926-4319-4e9d-8788-31f7ec35fbba\" (UID: \"1c0c7926-4319-4e9d-8788-31f7ec35fbba\") "
Jul 17 12:40:32.554719 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:32.554579    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c0c7926-4319-4e9d-8788-31f7ec35fbba-bundle\") pod \"1c0c7926-4319-4e9d-8788-31f7ec35fbba\" (UID: \"1c0c7926-4319-4e9d-8788-31f7ec35fbba\") "
Jul 17 12:40:32.555425 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:32.555400    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0c7926-4319-4e9d-8788-31f7ec35fbba-bundle" (OuterVolumeSpecName: "bundle") pod "1c0c7926-4319-4e9d-8788-31f7ec35fbba" (UID: "1c0c7926-4319-4e9d-8788-31f7ec35fbba"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:40:32.557058 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:32.557026    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0c7926-4319-4e9d-8788-31f7ec35fbba-kube-api-access-zvsfn" (OuterVolumeSpecName: "kube-api-access-zvsfn") pod "1c0c7926-4319-4e9d-8788-31f7ec35fbba" (UID: "1c0c7926-4319-4e9d-8788-31f7ec35fbba"). InnerVolumeSpecName "kube-api-access-zvsfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:40:32.560125 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:32.560100    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0c7926-4319-4e9d-8788-31f7ec35fbba-util" (OuterVolumeSpecName: "util") pod "1c0c7926-4319-4e9d-8788-31f7ec35fbba" (UID: "1c0c7926-4319-4e9d-8788-31f7ec35fbba"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:40:32.656131 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:32.656041    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c0c7926-4319-4e9d-8788-31f7ec35fbba-util\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:40:32.656131 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:32.656070    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvsfn\" (UniqueName: \"kubernetes.io/projected/1c0c7926-4319-4e9d-8788-31f7ec35fbba-kube-api-access-zvsfn\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:40:32.656131 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:32.656081    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c0c7926-4319-4e9d-8788-31f7ec35fbba-bundle\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:40:33.324066 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:33.324030    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78" event={"ID":"1c0c7926-4319-4e9d-8788-31f7ec35fbba","Type":"ContainerDied","Data":"aa1751ce35a06c23ea9802fc736e02fb27cf2b2b895a0ab2d4a61c171220ce94"}
Jul 17 12:40:33.324066 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:33.324069    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s5b78"
Jul 17 12:40:33.324283 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:33.324103    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa1751ce35a06c23ea9802fc736e02fb27cf2b2b895a0ab2d4a61c171220ce94"
Jul 17 12:40:38.427061 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:38.427022    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:39.811824 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:39.811785    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:40.340897 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:40.340863    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-krhvv"]
Jul 17 12:40:40.341163 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:40.341151    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0c7926-4319-4e9d-8788-31f7ec35fbba" containerName="util"
Jul 17 12:40:40.341213 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:40.341165    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0c7926-4319-4e9d-8788-31f7ec35fbba" containerName="util"
Jul 17 12:40:40.341213 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:40.341173    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0c7926-4319-4e9d-8788-31f7ec35fbba" containerName="pull"
Jul 17 12:40:40.341213 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:40.341179    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0c7926-4319-4e9d-8788-31f7ec35fbba" containerName="pull"
Jul 17 12:40:40.341213 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:40.341195    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0c7926-4319-4e9d-8788-31f7ec35fbba" containerName="extract"
Jul 17 12:40:40.341213 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:40.341200    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0c7926-4319-4e9d-8788-31f7ec35fbba" containerName="extract"
Jul 17 12:40:40.341353 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:40.341248    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c0c7926-4319-4e9d-8788-31f7ec35fbba" containerName="extract"
Jul 17 12:40:40.344175 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:40.344159    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-krhvv"
Jul 17 12:40:40.347939 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:40.347914    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 17 12:40:40.348081 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:40.347969    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-7rctl\""
Jul 17 12:40:40.348214 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:40.348196    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 17 12:40:40.360290 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:40.360262    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-krhvv"]
Jul 17 12:40:40.513183 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:40.513152    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4d2752e-6d28-4427-bcaf-8569d953519b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-krhvv\" (UID: \"e4d2752e-6d28-4427-bcaf-8569d953519b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-krhvv"
Jul 17 12:40:40.513358 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:40.513195    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg7pb\" (UniqueName: \"kubernetes.io/projected/e4d2752e-6d28-4427-bcaf-8569d953519b-kube-api-access-wg7pb\") pod \"cert-manager-operator-controller-manager-55c444bf86-krhvv\" (UID: \"e4d2752e-6d28-4427-bcaf-8569d953519b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-krhvv"
Jul 17 12:40:40.614196 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:40.614121    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4d2752e-6d28-4427-bcaf-8569d953519b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-krhvv\" (UID: \"e4d2752e-6d28-4427-bcaf-8569d953519b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-krhvv"
Jul 17 12:40:40.614196 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:40.614168    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg7pb\" (UniqueName: \"kubernetes.io/projected/e4d2752e-6d28-4427-bcaf-8569d953519b-kube-api-access-wg7pb\") pod \"cert-manager-operator-controller-manager-55c444bf86-krhvv\" (UID: \"e4d2752e-6d28-4427-bcaf-8569d953519b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-krhvv"
Jul 17 12:40:40.614584 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:40.614565    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4d2752e-6d28-4427-bcaf-8569d953519b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-krhvv\" (UID: \"e4d2752e-6d28-4427-bcaf-8569d953519b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-krhvv"
Jul 17 12:40:40.622971 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:40.622938    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg7pb\" (UniqueName: \"kubernetes.io/projected/e4d2752e-6d28-4427-bcaf-8569d953519b-kube-api-access-wg7pb\") pod \"cert-manager-operator-controller-manager-55c444bf86-krhvv\" (UID: \"e4d2752e-6d28-4427-bcaf-8569d953519b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-krhvv"
Jul 17 12:40:40.653832 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:40.653775    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-krhvv"
Jul 17 12:40:40.793455 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:40.793334    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-krhvv"]
Jul 17 12:40:40.796260 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:40:40.796229    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d2752e_6d28_4427_bcaf_8569d953519b.slice/crio-677684bb1198d1462107a004fc309bbc3a31e3bef86f779f6181d22a2bd0a711 WatchSource:0}: Error finding container 677684bb1198d1462107a004fc309bbc3a31e3bef86f779f6181d22a2bd0a711: Status 404 returned error can't find the container with id 677684bb1198d1462107a004fc309bbc3a31e3bef86f779f6181d22a2bd0a711
Jul 17 12:40:41.356711 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:41.356672    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-krhvv" event={"ID":"e4d2752e-6d28-4427-bcaf-8569d953519b","Type":"ContainerStarted","Data":"677684bb1198d1462107a004fc309bbc3a31e3bef86f779f6181d22a2bd0a711"}
Jul 17 12:40:42.020038 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:42.019968    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:42.128950 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:42.128910    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:44.127331 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:44.127299    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:44.189143 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:44.189104    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:44.377583 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:44.377475    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-krhvv" event={"ID":"e4d2752e-6d28-4427-bcaf-8569d953519b","Type":"ContainerStarted","Data":"3026706275d5e961edb33da33d41189522162f0a678fc0d0fdcd28b25b065859"}
Jul 17 12:40:44.397011 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:44.396944    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-krhvv" podStartSLOduration=1.862368882 podStartE2EDuration="4.396928685s" totalImagesPullingTime="2.534559803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:40:44.395337783 +0000 UTC m=+1088.526722440" watchObservedRunningTime="2026-07-17 12:40:44.396928685 +0000 UTC m=+1088.528313350"
Jul 17 12:40:44.764653 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:44.764557    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:44.860743 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:44.860692    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:45.303912 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:45.303875    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf"]
Jul 17 12:40:45.307407 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:45.307391    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf"
Jul 17 12:40:45.310125 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:45.310100    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 17 12:40:45.310235 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:45.310158    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xmv68\""
Jul 17 12:40:45.311171 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:45.311155    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 17 12:40:45.316229 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:45.316209    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf"]
Jul 17 12:40:45.354154 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:45.354125    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6053ddbe-b4b3-442a-bba6-5265d8a7474b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf\" (UID: \"6053ddbe-b4b3-442a-bba6-5265d8a7474b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf"
Jul 17 12:40:45.354310 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:45.354159    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnq8t\" (UniqueName: \"kubernetes.io/projected/6053ddbe-b4b3-442a-bba6-5265d8a7474b-kube-api-access-bnq8t\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf\" (UID: \"6053ddbe-b4b3-442a-bba6-5265d8a7474b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf"
Jul 17 12:40:45.354310 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:45.354231    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6053ddbe-b4b3-442a-bba6-5265d8a7474b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf\" (UID: \"6053ddbe-b4b3-442a-bba6-5265d8a7474b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf"
Jul 17 12:40:45.454833 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:45.454796    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6053ddbe-b4b3-442a-bba6-5265d8a7474b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf\" (UID: \"6053ddbe-b4b3-442a-bba6-5265d8a7474b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf"
Jul 17 12:40:45.454833 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:45.454833    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnq8t\" (UniqueName: \"kubernetes.io/projected/6053ddbe-b4b3-442a-bba6-5265d8a7474b-kube-api-access-bnq8t\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf\" (UID: \"6053ddbe-b4b3-442a-bba6-5265d8a7474b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf"
Jul 17 12:40:45.455124 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:45.454885    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6053ddbe-b4b3-442a-bba6-5265d8a7474b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf\" (UID: \"6053ddbe-b4b3-442a-bba6-5265d8a7474b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf"
Jul 17 12:40:45.455291 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:45.455275    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6053ddbe-b4b3-442a-bba6-5265d8a7474b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf\" (UID: \"6053ddbe-b4b3-442a-bba6-5265d8a7474b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf"
Jul 17 12:40:45.455348 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:45.455275    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6053ddbe-b4b3-442a-bba6-5265d8a7474b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf\" (UID: \"6053ddbe-b4b3-442a-bba6-5265d8a7474b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf"
Jul 17 12:40:45.463083 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:45.463060    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnq8t\" (UniqueName: \"kubernetes.io/projected/6053ddbe-b4b3-442a-bba6-5265d8a7474b-kube-api-access-bnq8t\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf\" (UID: \"6053ddbe-b4b3-442a-bba6-5265d8a7474b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf"
Jul 17 12:40:45.616915 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:45.616827    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf"
Jul 17 12:40:45.747860 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:45.747830    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf"]
Jul 17 12:40:45.750591 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:40:45.750561    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6053ddbe_b4b3_442a_bba6_5265d8a7474b.slice/crio-e27998bfd3ca0862ffb9a70e6243484912b809d5999c188404360edf15adc24d WatchSource:0}: Error finding container e27998bfd3ca0862ffb9a70e6243484912b809d5999c188404360edf15adc24d: Status 404 returned error can't find the container with id e27998bfd3ca0862ffb9a70e6243484912b809d5999c188404360edf15adc24d
Jul 17 12:40:46.139419 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:46.139385    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:46.388276 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:46.388233    2565 generic.go:356] "Generic (PLEG): container finished" podID="6053ddbe-b4b3-442a-bba6-5265d8a7474b" containerID="635a02705c4e6dde7717bf89dc65a4abe68e90b7d517f9a381731a76eee2cf82" exitCode=0
Jul 17 12:40:46.388746 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:46.388288    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf" event={"ID":"6053ddbe-b4b3-442a-bba6-5265d8a7474b","Type":"ContainerDied","Data":"635a02705c4e6dde7717bf89dc65a4abe68e90b7d517f9a381731a76eee2cf82"}
Jul 17 12:40:46.388746 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:46.388330    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf" event={"ID":"6053ddbe-b4b3-442a-bba6-5265d8a7474b","Type":"ContainerStarted","Data":"e27998bfd3ca0862ffb9a70e6243484912b809d5999c188404360edf15adc24d"}
Jul 17 12:40:46.434140 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:46.434103    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fb7jw"]
Jul 17 12:40:46.437361 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:46.437344    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fb7jw"
Jul 17 12:40:46.440274 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:46.440247    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 17 12:40:46.440274 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:46.440264    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-pn4p4\""
Jul 17 12:40:46.440445 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:46.440308    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 17 12:40:46.446649 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:46.446626    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fb7jw"]
Jul 17 12:40:46.459838 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:46.459800    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f141d940-297c-4b36-a817-8d2300429ed0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fb7jw\" (UID: \"f141d940-297c-4b36-a817-8d2300429ed0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fb7jw"
Jul 17 12:40:46.460003 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:46.459838    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqj2v\" (UniqueName: \"kubernetes.io/projected/f141d940-297c-4b36-a817-8d2300429ed0-kube-api-access-rqj2v\") pod \"cert-manager-cainjector-559cdd8c87-fb7jw\" (UID: \"f141d940-297c-4b36-a817-8d2300429ed0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fb7jw"
Jul 17 12:40:46.528701 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:46.528676    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:46.560213 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:46.560182    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f141d940-297c-4b36-a817-8d2300429ed0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fb7jw\" (UID: \"f141d940-297c-4b36-a817-8d2300429ed0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fb7jw"
Jul 17 12:40:46.560466 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:46.560223    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqj2v\" (UniqueName: \"kubernetes.io/projected/f141d940-297c-4b36-a817-8d2300429ed0-kube-api-access-rqj2v\") pod \"cert-manager-cainjector-559cdd8c87-fb7jw\" (UID: \"f141d940-297c-4b36-a817-8d2300429ed0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fb7jw"
Jul 17 12:40:46.572195 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:46.572166    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f141d940-297c-4b36-a817-8d2300429ed0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-fb7jw\" (UID: \"f141d940-297c-4b36-a817-8d2300429ed0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fb7jw"
Jul 17 12:40:46.572383 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:46.572363    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqj2v\" (UniqueName: \"kubernetes.io/projected/f141d940-297c-4b36-a817-8d2300429ed0-kube-api-access-rqj2v\") pod \"cert-manager-cainjector-559cdd8c87-fb7jw\" (UID: \"f141d940-297c-4b36-a817-8d2300429ed0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-fb7jw"
Jul 17 12:40:46.761188 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:46.761089    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fb7jw"
Jul 17 12:40:46.902243 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:46.902096    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-fb7jw"]
Jul 17 12:40:46.904888 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:40:46.904861    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf141d940_297c_4b36_a817_8d2300429ed0.slice/crio-291823655ba1d1f673fb790b8e5846266ecdeed8d26f854be6671c2c6dff5d67 WatchSource:0}: Error finding container 291823655ba1d1f673fb790b8e5846266ecdeed8d26f854be6671c2c6dff5d67: Status 404 returned error can't find the container with id 291823655ba1d1f673fb790b8e5846266ecdeed8d26f854be6671c2c6dff5d67
Jul 17 12:40:47.393606 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:47.393568    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fb7jw" event={"ID":"f141d940-297c-4b36-a817-8d2300429ed0","Type":"ContainerStarted","Data":"291823655ba1d1f673fb790b8e5846266ecdeed8d26f854be6671c2c6dff5d67"}
Jul 17 12:40:48.805242 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:48.805175    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:48.864696 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:48.864658    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:49.405597 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:49.405563    2565 generic.go:356] "Generic (PLEG): container finished" podID="6053ddbe-b4b3-442a-bba6-5265d8a7474b" containerID="bdce838c7677f43e72dd72a777712f5e11e9185c0a8becec454dcdc48df9b5ac" exitCode=0
Jul 17 12:40:49.405774 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:49.405664    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf" event={"ID":"6053ddbe-b4b3-442a-bba6-5265d8a7474b","Type":"ContainerDied","Data":"bdce838c7677f43e72dd72a777712f5e11e9185c0a8becec454dcdc48df9b5ac"}
Jul 17 12:40:50.412655 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:50.412618    2565 generic.go:356] "Generic (PLEG): container finished" podID="6053ddbe-b4b3-442a-bba6-5265d8a7474b" containerID="0121fc6e22cdfd58f0d16f77b383b9b7d075ee95ec92a3bde0c355afcc7d8846" exitCode=0
Jul 17 12:40:50.413137 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:50.412699    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf" event={"ID":"6053ddbe-b4b3-442a-bba6-5265d8a7474b","Type":"ContainerDied","Data":"0121fc6e22cdfd58f0d16f77b383b9b7d075ee95ec92a3bde0c355afcc7d8846"}
Jul 17 12:40:51.418288 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:51.418187    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fb7jw" event={"ID":"f141d940-297c-4b36-a817-8d2300429ed0","Type":"ContainerStarted","Data":"5ebf7ade09288f71a7d805e12dc5b50af999e275775d06377443e66b8c8ddd1f"}
Jul 17 12:40:51.434654 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:51.434597    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-fb7jw" podStartSLOduration=1.2601537139999999 podStartE2EDuration="5.434580301s" totalImagesPullingTime="4.174426587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:40:51.433459629 +0000 UTC m=+1095.564844296" watchObservedRunningTime="2026-07-17 12:40:51.434580301 +0000 UTC m=+1095.565964961"
Jul 17 12:40:51.570683 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:51.570646    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf"
Jul 17 12:40:51.598032 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:51.597973    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnq8t\" (UniqueName: \"kubernetes.io/projected/6053ddbe-b4b3-442a-bba6-5265d8a7474b-kube-api-access-bnq8t\") pod \"6053ddbe-b4b3-442a-bba6-5265d8a7474b\" (UID: \"6053ddbe-b4b3-442a-bba6-5265d8a7474b\") "
Jul 17 12:40:51.598213 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:51.598068    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6053ddbe-b4b3-442a-bba6-5265d8a7474b-bundle\") pod \"6053ddbe-b4b3-442a-bba6-5265d8a7474b\" (UID: \"6053ddbe-b4b3-442a-bba6-5265d8a7474b\") "
Jul 17 12:40:51.598213 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:51.598121    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6053ddbe-b4b3-442a-bba6-5265d8a7474b-util\") pod \"6053ddbe-b4b3-442a-bba6-5265d8a7474b\" (UID: \"6053ddbe-b4b3-442a-bba6-5265d8a7474b\") "
Jul 17 12:40:51.598474 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:51.598449    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6053ddbe-b4b3-442a-bba6-5265d8a7474b-bundle" (OuterVolumeSpecName: "bundle") pod "6053ddbe-b4b3-442a-bba6-5265d8a7474b" (UID: "6053ddbe-b4b3-442a-bba6-5265d8a7474b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:40:51.600346 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:51.600315    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6053ddbe-b4b3-442a-bba6-5265d8a7474b-kube-api-access-bnq8t" (OuterVolumeSpecName: "kube-api-access-bnq8t") pod "6053ddbe-b4b3-442a-bba6-5265d8a7474b" (UID: "6053ddbe-b4b3-442a-bba6-5265d8a7474b"). InnerVolumeSpecName "kube-api-access-bnq8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:40:51.699086 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:51.699012    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnq8t\" (UniqueName: \"kubernetes.io/projected/6053ddbe-b4b3-442a-bba6-5265d8a7474b-kube-api-access-bnq8t\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:40:51.699086 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:51.699037    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6053ddbe-b4b3-442a-bba6-5265d8a7474b-bundle\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:40:51.931546 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:51.931513    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:51.950698 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:51.950613    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6053ddbe-b4b3-442a-bba6-5265d8a7474b-util" (OuterVolumeSpecName: "util") pod "6053ddbe-b4b3-442a-bba6-5265d8a7474b" (UID: "6053ddbe-b4b3-442a-bba6-5265d8a7474b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:40:52.002381 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:52.002350    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6053ddbe-b4b3-442a-bba6-5265d8a7474b-util\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:40:52.424436 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:52.424406    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf"
Jul 17 12:40:52.424436 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:52.424428    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fd6spf" event={"ID":"6053ddbe-b4b3-442a-bba6-5265d8a7474b","Type":"ContainerDied","Data":"e27998bfd3ca0862ffb9a70e6243484912b809d5999c188404360edf15adc24d"}
Jul 17 12:40:52.424974 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:52.424476    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e27998bfd3ca0862ffb9a70e6243484912b809d5999c188404360edf15adc24d"
Jul 17 12:40:53.187214 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.187158    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:53.773051 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.773019    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6554dd577b-4fgjf"]
Jul 17 12:40:53.773442 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.773359    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6053ddbe-b4b3-442a-bba6-5265d8a7474b" containerName="util"
Jul 17 12:40:53.773442 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.773378    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6053ddbe-b4b3-442a-bba6-5265d8a7474b" containerName="util"
Jul 17 12:40:53.773442 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.773396    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6053ddbe-b4b3-442a-bba6-5265d8a7474b" containerName="pull"
Jul 17 12:40:53.773442 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.773401    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6053ddbe-b4b3-442a-bba6-5265d8a7474b" containerName="pull"
Jul 17 12:40:53.773442 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.773410    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6053ddbe-b4b3-442a-bba6-5265d8a7474b" containerName="extract"
Jul 17 12:40:53.773442 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.773415    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6053ddbe-b4b3-442a-bba6-5265d8a7474b" containerName="extract"
Jul 17 12:40:53.773622 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.773478    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6053ddbe-b4b3-442a-bba6-5265d8a7474b" containerName="extract"
Jul 17 12:40:53.776863 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.776842    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6554dd577b-4fgjf"
Jul 17 12:40:53.780221 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.780179    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:53.780715 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.780694    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-24hq6\""
Jul 17 12:40:53.780831 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.780762    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 17 12:40:53.780831 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.780775    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 17 12:40:53.780947 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.780725    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 17 12:40:53.780947 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.780743    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 17 12:40:53.783902 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.783868    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6554dd577b-4fgjf"]
Jul 17 12:40:53.816660 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.816624    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldmnj\" (UniqueName: \"kubernetes.io/projected/8c119593-16f7-4ca1-9a76-a3bb481d9237-kube-api-access-ldmnj\") pod \"managed-serviceaccount-addon-agent-6554dd577b-4fgjf\" (UID: \"8c119593-16f7-4ca1-9a76-a3bb481d9237\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6554dd577b-4fgjf"
Jul 17 12:40:53.816840 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.816682    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8c119593-16f7-4ca1-9a76-a3bb481d9237-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6554dd577b-4fgjf\" (UID: \"8c119593-16f7-4ca1-9a76-a3bb481d9237\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6554dd577b-4fgjf"
Jul 17 12:40:53.879146 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.879108    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:53.897384 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.897343    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f88b5d45-p4zwx"]
Jul 17 12:40:53.899549 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.899531    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f88b5d45-p4zwx"
Jul 17 12:40:53.902713 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.902698    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 17 12:40:53.912827 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.912803    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f88b5d45-p4zwx"]
Jul 17 12:40:53.917537 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.917514    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dd4734cc-8fa6-4ec8-b9db-a362eb1c8112-klusterlet-config\") pod \"klusterlet-addon-workmgr-76f88b5d45-p4zwx\" (UID: \"dd4734cc-8fa6-4ec8-b9db-a362eb1c8112\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f88b5d45-p4zwx"
Jul 17 12:40:53.917636 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.917558    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldmnj\" (UniqueName: \"kubernetes.io/projected/8c119593-16f7-4ca1-9a76-a3bb481d9237-kube-api-access-ldmnj\") pod \"managed-serviceaccount-addon-agent-6554dd577b-4fgjf\" (UID: \"8c119593-16f7-4ca1-9a76-a3bb481d9237\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6554dd577b-4fgjf"
Jul 17 12:40:53.917636 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.917577    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgh6x\" (UniqueName: \"kubernetes.io/projected/dd4734cc-8fa6-4ec8-b9db-a362eb1c8112-kube-api-access-hgh6x\") pod \"klusterlet-addon-workmgr-76f88b5d45-p4zwx\" (UID: \"dd4734cc-8fa6-4ec8-b9db-a362eb1c8112\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f88b5d45-p4zwx"
Jul 17 12:40:53.917725 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.917712    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd4734cc-8fa6-4ec8-b9db-a362eb1c8112-tmp\") pod \"klusterlet-addon-workmgr-76f88b5d45-p4zwx\" (UID: \"dd4734cc-8fa6-4ec8-b9db-a362eb1c8112\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f88b5d45-p4zwx"
Jul 17 12:40:53.917760 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.917742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8c119593-16f7-4ca1-9a76-a3bb481d9237-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6554dd577b-4fgjf\" (UID: \"8c119593-16f7-4ca1-9a76-a3bb481d9237\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6554dd577b-4fgjf"
Jul 17 12:40:53.920348 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.920323    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8c119593-16f7-4ca1-9a76-a3bb481d9237-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6554dd577b-4fgjf\" (UID: \"8c119593-16f7-4ca1-9a76-a3bb481d9237\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6554dd577b-4fgjf"
Jul 17 12:40:53.929047 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.929025    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldmnj\" (UniqueName: \"kubernetes.io/projected/8c119593-16f7-4ca1-9a76-a3bb481d9237-kube-api-access-ldmnj\") pod \"managed-serviceaccount-addon-agent-6554dd577b-4fgjf\" (UID: \"8c119593-16f7-4ca1-9a76-a3bb481d9237\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6554dd577b-4fgjf"
Jul 17 12:40:53.954735 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:53.954698    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:54.018761 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:54.018723    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dd4734cc-8fa6-4ec8-b9db-a362eb1c8112-klusterlet-config\") pod \"klusterlet-addon-workmgr-76f88b5d45-p4zwx\" (UID: \"dd4734cc-8fa6-4ec8-b9db-a362eb1c8112\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f88b5d45-p4zwx"
Jul 17 12:40:54.018930 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:54.018782    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgh6x\" (UniqueName: \"kubernetes.io/projected/dd4734cc-8fa6-4ec8-b9db-a362eb1c8112-kube-api-access-hgh6x\") pod \"klusterlet-addon-workmgr-76f88b5d45-p4zwx\" (UID: \"dd4734cc-8fa6-4ec8-b9db-a362eb1c8112\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f88b5d45-p4zwx"
Jul 17 12:40:54.018930 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:54.018847    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd4734cc-8fa6-4ec8-b9db-a362eb1c8112-tmp\") pod \"klusterlet-addon-workmgr-76f88b5d45-p4zwx\" (UID: \"dd4734cc-8fa6-4ec8-b9db-a362eb1c8112\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f88b5d45-p4zwx"
Jul 17 12:40:54.019353 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:54.019334    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd4734cc-8fa6-4ec8-b9db-a362eb1c8112-tmp\") pod \"klusterlet-addon-workmgr-76f88b5d45-p4zwx\" (UID: \"dd4734cc-8fa6-4ec8-b9db-a362eb1c8112\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f88b5d45-p4zwx"
Jul 17 12:40:54.021489 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:54.021472    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dd4734cc-8fa6-4ec8-b9db-a362eb1c8112-klusterlet-config\") pod \"klusterlet-addon-workmgr-76f88b5d45-p4zwx\" (UID: \"dd4734cc-8fa6-4ec8-b9db-a362eb1c8112\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f88b5d45-p4zwx"
Jul 17 12:40:54.027051 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:54.026973    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgh6x\" (UniqueName: \"kubernetes.io/projected/dd4734cc-8fa6-4ec8-b9db-a362eb1c8112-kube-api-access-hgh6x\") pod \"klusterlet-addon-workmgr-76f88b5d45-p4zwx\" (UID: \"dd4734cc-8fa6-4ec8-b9db-a362eb1c8112\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f88b5d45-p4zwx"
Jul 17 12:40:54.087756 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:54.087721    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6554dd577b-4fgjf"
Jul 17 12:40:54.209266 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:54.209232    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f88b5d45-p4zwx"
Jul 17 12:40:54.223114 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:54.223086    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6554dd577b-4fgjf"]
Jul 17 12:40:54.225096 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:40:54.225062    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c119593_16f7_4ca1_9a76_a3bb481d9237.slice/crio-1609d752b5c397735aa0a65c3fc1c84228f2549b64c71fc400c5e3dd31ea1eba WatchSource:0}: Error finding container 1609d752b5c397735aa0a65c3fc1c84228f2549b64c71fc400c5e3dd31ea1eba: Status 404 returned error can't find the container with id 1609d752b5c397735aa0a65c3fc1c84228f2549b64c71fc400c5e3dd31ea1eba
Jul 17 12:40:54.347012 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:54.346968    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f88b5d45-p4zwx"]
Jul 17 12:40:54.349192 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:40:54.349164    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4734cc_8fa6_4ec8_b9db_a362eb1c8112.slice/crio-45e6c354010bd7a12d5ae0f80d7f7a966f73892571cb684336e2d134815e1054 WatchSource:0}: Error finding container 45e6c354010bd7a12d5ae0f80d7f7a966f73892571cb684336e2d134815e1054: Status 404 returned error can't find the container with id 45e6c354010bd7a12d5ae0f80d7f7a966f73892571cb684336e2d134815e1054
Jul 17 12:40:54.434934 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:54.434889    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f88b5d45-p4zwx" event={"ID":"dd4734cc-8fa6-4ec8-b9db-a362eb1c8112","Type":"ContainerStarted","Data":"45e6c354010bd7a12d5ae0f80d7f7a966f73892571cb684336e2d134815e1054"}
Jul 17 12:40:54.435946 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:54.435924    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6554dd577b-4fgjf" event={"ID":"8c119593-16f7-4ca1-9a76-a3bb481d9237","Type":"ContainerStarted","Data":"1609d752b5c397735aa0a65c3fc1c84228f2549b64c71fc400c5e3dd31ea1eba"}
Jul 17 12:40:54.962079 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:54.962046    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:56.260731 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:56.260698    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:56.361580 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:56.361547    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:40:58.242827 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:58.242794    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk64p"]
Jul 17 12:40:58.245921 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:58.245902    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk64p"
Jul 17 12:40:58.248835 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:58.248811    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-cm6bz\""
Jul 17 12:40:58.248977 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:58.248811    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 17 12:40:58.248977 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:58.248817    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 17 12:40:58.255122 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:58.255097    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk64p"]
Jul 17 12:40:58.352145 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:58.352107    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rrxn\" (UniqueName: \"kubernetes.io/projected/5f7f96d6-0e5f-4da3-8fea-db93ff820da5-kube-api-access-7rrxn\") pod \"openshift-lws-operator-fd8ccff4c-fk64p\" (UID: \"5f7f96d6-0e5f-4da3-8fea-db93ff820da5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk64p"
Jul 17 12:40:58.352325 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:58.352194    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f7f96d6-0e5f-4da3-8fea-db93ff820da5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fk64p\" (UID: \"5f7f96d6-0e5f-4da3-8fea-db93ff820da5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk64p"
Jul 17 12:40:58.452793 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:58.452763    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f7f96d6-0e5f-4da3-8fea-db93ff820da5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fk64p\" (UID: \"5f7f96d6-0e5f-4da3-8fea-db93ff820da5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk64p"
Jul 17 12:40:58.452972 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:58.452808    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rrxn\" (UniqueName: \"kubernetes.io/projected/5f7f96d6-0e5f-4da3-8fea-db93ff820da5-kube-api-access-7rrxn\") pod \"openshift-lws-operator-fd8ccff4c-fk64p\" (UID: \"5f7f96d6-0e5f-4da3-8fea-db93ff820da5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk64p"
Jul 17 12:40:58.453169 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:58.453152    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f7f96d6-0e5f-4da3-8fea-db93ff820da5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fk64p\" (UID: \"5f7f96d6-0e5f-4da3-8fea-db93ff820da5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk64p"
Jul 17 12:40:58.469273 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:58.469239    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rrxn\" (UniqueName: \"kubernetes.io/projected/5f7f96d6-0e5f-4da3-8fea-db93ff820da5-kube-api-access-7rrxn\") pod \"openshift-lws-operator-fd8ccff4c-fk64p\" (UID: \"5f7f96d6-0e5f-4da3-8fea-db93ff820da5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk64p"
Jul 17 12:40:58.555384 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:58.555348    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk64p"
Jul 17 12:40:58.688067 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:58.688039    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk64p"]
Jul 17 12:40:58.690491 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:40:58.690463    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f7f96d6_0e5f_4da3_8fea_db93ff820da5.slice/crio-431cfaec53a5ad011bcaa7deac7a895dcc9587e4d4ba88908ec71ef6b3edafa4 WatchSource:0}: Error finding container 431cfaec53a5ad011bcaa7deac7a895dcc9587e4d4ba88908ec71ef6b3edafa4: Status 404 returned error can't find the container with id 431cfaec53a5ad011bcaa7deac7a895dcc9587e4d4ba88908ec71ef6b3edafa4
Jul 17 12:40:59.460750 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:40:59.460702    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk64p" event={"ID":"5f7f96d6-0e5f-4da3-8fea-db93ff820da5","Type":"ContainerStarted","Data":"431cfaec53a5ad011bcaa7deac7a895dcc9587e4d4ba88908ec71ef6b3edafa4"}
Jul 17 12:41:02.477383 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:02.477340    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6554dd577b-4fgjf" event={"ID":"8c119593-16f7-4ca1-9a76-a3bb481d9237","Type":"ContainerStarted","Data":"de7dae66a78d93ea783ae1eab934fc7fd83fa16873a74692c5e7b77c2055e12c"}
Jul 17 12:41:02.499668 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:02.499580    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6554dd577b-4fgjf" podStartSLOduration=2.154037891 podStartE2EDuration="9.499562697s" totalImagesPullingTime="7.345524806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:41:02.498480897 +0000 UTC m=+1106.629865558" watchObservedRunningTime="2026-07-17 12:41:02.499562697 +0000 UTC m=+1106.630947357"
Jul 17 12:41:03.487104 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:03.487062    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk64p" event={"ID":"5f7f96d6-0e5f-4da3-8fea-db93ff820da5","Type":"ContainerStarted","Data":"8bd42a428ebff23fc9fead9bff75c46dd4c1eb2757bbcf57ec28422979612de2"}
Jul 17 12:41:03.510812 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:03.510720    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk64p" podStartSLOduration=1.312569609 podStartE2EDuration="5.510700759s" totalImagesPullingTime="4.19813115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:40:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:41:03.510193393 +0000 UTC m=+1107.641578055" watchObservedRunningTime="2026-07-17 12:41:03.510700759 +0000 UTC m=+1107.642085421"
Jul 17 12:41:04.379380 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:04.379348    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:04.477845 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:04.477800    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:06.065055 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:06.065016    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:06.292006 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:06.291965    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v"]
Jul 17 12:41:06.294630 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:06.294608    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v"
Jul 17 12:41:06.297126 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:06.297104    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 17 12:41:06.297126 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:06.297116    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xmv68\""
Jul 17 12:41:06.297274 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:06.297104    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 17 12:41:06.304670 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:06.304643    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v"]
Jul 17 12:41:06.422641 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:06.422547    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6jhh\" (UniqueName: \"kubernetes.io/projected/6abd9a2c-60fd-4b8f-aefa-486991a0d447-kube-api-access-b6jhh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v\" (UID: \"6abd9a2c-60fd-4b8f-aefa-486991a0d447\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v"
Jul 17 12:41:06.422641 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:06.422619    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6abd9a2c-60fd-4b8f-aefa-486991a0d447-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v\" (UID: \"6abd9a2c-60fd-4b8f-aefa-486991a0d447\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v"
Jul 17 12:41:06.422842 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:06.422657    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6abd9a2c-60fd-4b8f-aefa-486991a0d447-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v\" (UID: \"6abd9a2c-60fd-4b8f-aefa-486991a0d447\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v"
Jul 17 12:41:06.502766 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:06.502727    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f88b5d45-p4zwx" event={"ID":"dd4734cc-8fa6-4ec8-b9db-a362eb1c8112","Type":"ContainerStarted","Data":"3c4267886d16350cfee36699eaccf94f2a9bcdca0327299b9fb9bdbacb4ac564"}
Jul 17 12:41:06.502951 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:06.502937    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f88b5d45-p4zwx"
Jul 17 12:41:06.504689 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:06.504664    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f88b5d45-p4zwx"
Jul 17 12:41:06.517764 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:06.517683    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-76f88b5d45-p4zwx" podStartSLOduration=1.7680469460000001 podStartE2EDuration="13.517664701s" totalImagesPullingTime="11.749617755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:41:06.517559396 +0000 UTC m=+1110.648944070" watchObservedRunningTime="2026-07-17 12:41:06.517664701 +0000 UTC m=+1110.649049361"
Jul 17 12:41:06.523218 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:06.523189    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6abd9a2c-60fd-4b8f-aefa-486991a0d447-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v\" (UID: \"6abd9a2c-60fd-4b8f-aefa-486991a0d447\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v"
Jul 17 12:41:06.523368 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:06.523226    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6abd9a2c-60fd-4b8f-aefa-486991a0d447-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v\" (UID: \"6abd9a2c-60fd-4b8f-aefa-486991a0d447\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v"
Jul 17 12:41:06.523368 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:06.523275    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6jhh\" (UniqueName: \"kubernetes.io/projected/6abd9a2c-60fd-4b8f-aefa-486991a0d447-kube-api-access-b6jhh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v\" (UID: \"6abd9a2c-60fd-4b8f-aefa-486991a0d447\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v"
Jul 17 12:41:06.523704 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:06.523683    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6abd9a2c-60fd-4b8f-aefa-486991a0d447-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v\" (UID: \"6abd9a2c-60fd-4b8f-aefa-486991a0d447\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v"
Jul 17 12:41:06.523745 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:06.523683    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6abd9a2c-60fd-4b8f-aefa-486991a0d447-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v\" (UID: \"6abd9a2c-60fd-4b8f-aefa-486991a0d447\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v"
Jul 17 12:41:06.530826 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:06.530797    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6jhh\" (UniqueName: \"kubernetes.io/projected/6abd9a2c-60fd-4b8f-aefa-486991a0d447-kube-api-access-b6jhh\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v\" (UID: \"6abd9a2c-60fd-4b8f-aefa-486991a0d447\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v"
Jul 17 12:41:06.604910 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:06.604878    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v"
Jul 17 12:41:06.755919 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:06.755777    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v"]
Jul 17 12:41:06.758222 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:41:06.758189    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abd9a2c_60fd_4b8f_aefa_486991a0d447.slice/crio-6b3b2c3c676239e0321114048920d5181f5b5a18e95a3d9d642727f2f3671fa6 WatchSource:0}: Error finding container 6b3b2c3c676239e0321114048920d5181f5b5a18e95a3d9d642727f2f3671fa6: Status 404 returned error can't find the container with id 6b3b2c3c676239e0321114048920d5181f5b5a18e95a3d9d642727f2f3671fa6
Jul 17 12:41:07.508453 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:07.508416    2565 generic.go:356] "Generic (PLEG): container finished" podID="6abd9a2c-60fd-4b8f-aefa-486991a0d447" containerID="daf2ff084baf9eae7e64563d45fb3a84c0b6cbec0da541084fe6abde7c3488ef" exitCode=0
Jul 17 12:41:07.508859 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:07.508502    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v" event={"ID":"6abd9a2c-60fd-4b8f-aefa-486991a0d447","Type":"ContainerDied","Data":"daf2ff084baf9eae7e64563d45fb3a84c0b6cbec0da541084fe6abde7c3488ef"}
Jul 17 12:41:07.508859 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:07.508553    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v" event={"ID":"6abd9a2c-60fd-4b8f-aefa-486991a0d447","Type":"ContainerStarted","Data":"6b3b2c3c676239e0321114048920d5181f5b5a18e95a3d9d642727f2f3671fa6"}
Jul 17 12:41:08.514577 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:08.514477    2565 generic.go:356] "Generic (PLEG): container finished" podID="6abd9a2c-60fd-4b8f-aefa-486991a0d447" containerID="8651aeaa75a44139ee510a032fb94c912fc3a79d3f8c3f23ed41ad2c51c0eeec" exitCode=0
Jul 17 12:41:08.515062 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:08.514566    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v" event={"ID":"6abd9a2c-60fd-4b8f-aefa-486991a0d447","Type":"ContainerDied","Data":"8651aeaa75a44139ee510a032fb94c912fc3a79d3f8c3f23ed41ad2c51c0eeec"}
Jul 17 12:41:09.521361 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:09.521327    2565 generic.go:356] "Generic (PLEG): container finished" podID="6abd9a2c-60fd-4b8f-aefa-486991a0d447" containerID="954a11a63e6ca62156d70e9bd45ec65ae0894683d0db1157aa0e2bd209717407" exitCode=0
Jul 17 12:41:09.521745 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:09.521404    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v" event={"ID":"6abd9a2c-60fd-4b8f-aefa-486991a0d447","Type":"ContainerDied","Data":"954a11a63e6ca62156d70e9bd45ec65ae0894683d0db1157aa0e2bd209717407"}
Jul 17 12:41:10.648263 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:10.648241    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v"
Jul 17 12:41:10.760156 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:10.760122    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6abd9a2c-60fd-4b8f-aefa-486991a0d447-bundle\") pod \"6abd9a2c-60fd-4b8f-aefa-486991a0d447\" (UID: \"6abd9a2c-60fd-4b8f-aefa-486991a0d447\") "
Jul 17 12:41:10.760316 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:10.760172    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6jhh\" (UniqueName: \"kubernetes.io/projected/6abd9a2c-60fd-4b8f-aefa-486991a0d447-kube-api-access-b6jhh\") pod \"6abd9a2c-60fd-4b8f-aefa-486991a0d447\" (UID: \"6abd9a2c-60fd-4b8f-aefa-486991a0d447\") "
Jul 17 12:41:10.760316 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:10.760212    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6abd9a2c-60fd-4b8f-aefa-486991a0d447-util\") pod \"6abd9a2c-60fd-4b8f-aefa-486991a0d447\" (UID: \"6abd9a2c-60fd-4b8f-aefa-486991a0d447\") "
Jul 17 12:41:10.760878 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:10.760800    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abd9a2c-60fd-4b8f-aefa-486991a0d447-bundle" (OuterVolumeSpecName: "bundle") pod "6abd9a2c-60fd-4b8f-aefa-486991a0d447" (UID: "6abd9a2c-60fd-4b8f-aefa-486991a0d447"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:41:10.762399 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:10.762376    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6abd9a2c-60fd-4b8f-aefa-486991a0d447-kube-api-access-b6jhh" (OuterVolumeSpecName: "kube-api-access-b6jhh") pod "6abd9a2c-60fd-4b8f-aefa-486991a0d447" (UID: "6abd9a2c-60fd-4b8f-aefa-486991a0d447"). InnerVolumeSpecName "kube-api-access-b6jhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:41:10.767840 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:10.767817    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abd9a2c-60fd-4b8f-aefa-486991a0d447-util" (OuterVolumeSpecName: "util") pod "6abd9a2c-60fd-4b8f-aefa-486991a0d447" (UID: "6abd9a2c-60fd-4b8f-aefa-486991a0d447"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:41:10.861418 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:10.861392    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6abd9a2c-60fd-4b8f-aefa-486991a0d447-util\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:41:10.861418 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:10.861416    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6abd9a2c-60fd-4b8f-aefa-486991a0d447-bundle\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:41:10.861581 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:10.861425    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6jhh\" (UniqueName: \"kubernetes.io/projected/6abd9a2c-60fd-4b8f-aefa-486991a0d447-kube-api-access-b6jhh\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:41:11.531804 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:11.531778    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v"
Jul 17 12:41:11.531968 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:11.531776    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afpb66v" event={"ID":"6abd9a2c-60fd-4b8f-aefa-486991a0d447","Type":"ContainerDied","Data":"6b3b2c3c676239e0321114048920d5181f5b5a18e95a3d9d642727f2f3671fa6"}
Jul 17 12:41:11.531968 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:11.531894    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b3b2c3c676239e0321114048920d5181f5b5a18e95a3d9d642727f2f3671fa6"
Jul 17 12:41:23.221618 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:23.221578    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:23.323417 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:23.323381    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:23.875631 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:23.875594    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:24.103841 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.103796    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:24.209289 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.209193    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:24.245974 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.245934    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs"]
Jul 17 12:41:24.246444 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.246428    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6abd9a2c-60fd-4b8f-aefa-486991a0d447" containerName="util"
Jul 17 12:41:24.246509 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.246447    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6abd9a2c-60fd-4b8f-aefa-486991a0d447" containerName="util"
Jul 17 12:41:24.246509 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.246488    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6abd9a2c-60fd-4b8f-aefa-486991a0d447" containerName="extract"
Jul 17 12:41:24.246509 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.246497    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6abd9a2c-60fd-4b8f-aefa-486991a0d447" containerName="extract"
Jul 17 12:41:24.246722 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.246510    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6abd9a2c-60fd-4b8f-aefa-486991a0d447" containerName="pull"
Jul 17 12:41:24.246722 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.246519    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6abd9a2c-60fd-4b8f-aefa-486991a0d447" containerName="pull"
Jul 17 12:41:24.246722 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.246600    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6abd9a2c-60fd-4b8f-aefa-486991a0d447" containerName="extract"
Jul 17 12:41:24.289212 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.289170    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs"]
Jul 17 12:41:24.289413 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.289335    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs"
Jul 17 12:41:24.293506 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.293472    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 17 12:41:24.293819 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.293798    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 17 12:41:24.294241 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.294211    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 17 12:41:24.294750 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.294731    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 17 12:41:24.295038 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.295014    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-p95ns\""
Jul 17 12:41:24.379721 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.379687    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47x2s\" (UniqueName: \"kubernetes.io/projected/d039365a-014f-4608-879d-63fcd59f19b3-kube-api-access-47x2s\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-4prbs\" (UID: \"d039365a-014f-4608-879d-63fcd59f19b3\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs"
Jul 17 12:41:24.379933 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.379760    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d039365a-014f-4608-879d-63fcd59f19b3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-4prbs\" (UID: \"d039365a-014f-4608-879d-63fcd59f19b3\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs"
Jul 17 12:41:24.379933 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.379816    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d039365a-014f-4608-879d-63fcd59f19b3-tmp\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-4prbs\" (UID: \"d039365a-014f-4608-879d-63fcd59f19b3\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs"
Jul 17 12:41:24.379933 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.379881    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d039365a-014f-4608-879d-63fcd59f19b3-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-4prbs\" (UID: \"d039365a-014f-4608-879d-63fcd59f19b3\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs"
Jul 17 12:41:24.379933 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.379916    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d039365a-014f-4608-879d-63fcd59f19b3-manifests\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-4prbs\" (UID: \"d039365a-014f-4608-879d-63fcd59f19b3\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs"
Jul 17 12:41:24.480915 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.480816    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47x2s\" (UniqueName: \"kubernetes.io/projected/d039365a-014f-4608-879d-63fcd59f19b3-kube-api-access-47x2s\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-4prbs\" (UID: \"d039365a-014f-4608-879d-63fcd59f19b3\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs"
Jul 17 12:41:24.480915 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.480886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d039365a-014f-4608-879d-63fcd59f19b3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-4prbs\" (UID: \"d039365a-014f-4608-879d-63fcd59f19b3\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs"
Jul 17 12:41:24.481197 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.480941    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d039365a-014f-4608-879d-63fcd59f19b3-tmp\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-4prbs\" (UID: \"d039365a-014f-4608-879d-63fcd59f19b3\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs"
Jul 17 12:41:24.481197 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.480976    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d039365a-014f-4608-879d-63fcd59f19b3-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-4prbs\" (UID: \"d039365a-014f-4608-879d-63fcd59f19b3\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs"
Jul 17 12:41:24.481197 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.481029    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d039365a-014f-4608-879d-63fcd59f19b3-manifests\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-4prbs\" (UID: \"d039365a-014f-4608-879d-63fcd59f19b3\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs"
Jul 17 12:41:24.481507 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.481475    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d039365a-014f-4608-879d-63fcd59f19b3-manifests\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-4prbs\" (UID: \"d039365a-014f-4608-879d-63fcd59f19b3\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs"
Jul 17 12:41:24.483592 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.483572    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d039365a-014f-4608-879d-63fcd59f19b3-tmp\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-4prbs\" (UID: \"d039365a-014f-4608-879d-63fcd59f19b3\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs"
Jul 17 12:41:24.483743 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.483725    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d039365a-014f-4608-879d-63fcd59f19b3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-4prbs\" (UID: \"d039365a-014f-4608-879d-63fcd59f19b3\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs"
Jul 17 12:41:24.483786 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.483738    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d039365a-014f-4608-879d-63fcd59f19b3-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-4prbs\" (UID: \"d039365a-014f-4608-879d-63fcd59f19b3\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs"
Jul 17 12:41:24.492691 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.492658    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47x2s\" (UniqueName: \"kubernetes.io/projected/d039365a-014f-4608-879d-63fcd59f19b3-kube-api-access-47x2s\") pod \"opendatahub-operator-controller-manager-5d8fb4667b-4prbs\" (UID: \"d039365a-014f-4608-879d-63fcd59f19b3\") " pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs"
Jul 17 12:41:24.605727 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.605683    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs"
Jul 17 12:41:24.769591 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:24.769560    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs"]
Jul 17 12:41:24.772297 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:41:24.772271    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd039365a_014f_4608_879d_63fcd59f19b3.slice/crio-39ad393dc84863c057fd300423781be84a1aecd769731f48fd521d5e06400a14 WatchSource:0}: Error finding container 39ad393dc84863c057fd300423781be84a1aecd769731f48fd521d5e06400a14: Status 404 returned error can't find the container with id 39ad393dc84863c057fd300423781be84a1aecd769731f48fd521d5e06400a14
Jul 17 12:41:25.603792 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:25.603730    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs" event={"ID":"d039365a-014f-4608-879d-63fcd59f19b3","Type":"ContainerStarted","Data":"39ad393dc84863c057fd300423781be84a1aecd769731f48fd521d5e06400a14"}
Jul 17 12:41:26.413471 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:26.413424    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:26.514378 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:26.514324    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:28.620078 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:28.620041    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs" event={"ID":"d039365a-014f-4608-879d-63fcd59f19b3","Type":"ContainerStarted","Data":"b6820faff2ed94b978355bbc65dead7d19c0addc990268ff2903ed1a0aa96aa4"}
Jul 17 12:41:29.626336 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:29.626298    2565 generic.go:356] "Generic (PLEG): container finished" podID="d039365a-014f-4608-879d-63fcd59f19b3" containerID="b6820faff2ed94b978355bbc65dead7d19c0addc990268ff2903ed1a0aa96aa4" exitCode=0
Jul 17 12:41:29.626730 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:29.626357    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs" event={"ID":"d039365a-014f-4608-879d-63fcd59f19b3","Type":"ContainerDied","Data":"b6820faff2ed94b978355bbc65dead7d19c0addc990268ff2903ed1a0aa96aa4"}
Jul 17 12:41:30.633589 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:30.633549    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs" event={"ID":"d039365a-014f-4608-879d-63fcd59f19b3","Type":"ContainerStarted","Data":"a8368f2a6b61736ccd8013cf44016812c4bfd1826b73130bb6ebca212f426769"}
Jul 17 12:41:30.634015 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:30.633679    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs"
Jul 17 12:41:30.659102 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:30.659039    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs" podStartSLOduration=2.288291299 podStartE2EDuration="6.659013847s" totalImagesPullingTime="3.291515941s" totalInitContainerRuntime="1.079206607s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:41:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:41:30.656361791 +0000 UTC m=+1134.787746461" watchObservedRunningTime="2026-07-17 12:41:30.659013847 +0000 UTC m=+1134.790398508"
Jul 17 12:41:32.270078 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:32.270035    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:32.381257 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:32.381224    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:36.005888 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:36.005844    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:36.112461 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:36.112424    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:41.640380 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:41.640351    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5d8fb4667b-4prbs"
Jul 17 12:41:43.219320 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:43.219289    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:43.356267 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:43.356233    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:43.722681 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:43.722638    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6bfcdd9cc7-psxhp"]
Jul 17 12:41:43.729664 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:43.729636    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-psxhp"
Jul 17 12:41:43.733995 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:43.733954    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 17 12:41:43.734135 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:43.734023    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 17 12:41:43.734200 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:43.734141    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 17 12:41:43.734283 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:43.734267    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cx57d\""
Jul 17 12:41:43.734332 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:43.734308    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 17 12:41:43.741546 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:43.741522    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bfcdd9cc7-psxhp"]
Jul 17 12:41:43.747281 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:43.747256    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07-tls-certs\") pod \"kube-auth-proxy-6bfcdd9cc7-psxhp\" (UID: \"52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07\") " pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-psxhp"
Jul 17 12:41:43.747413 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:43.747307    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07-tmp\") pod \"kube-auth-proxy-6bfcdd9cc7-psxhp\" (UID: \"52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07\") " pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-psxhp"
Jul 17 12:41:43.747413 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:43.747365    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w5hr\" (UniqueName: \"kubernetes.io/projected/52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07-kube-api-access-8w5hr\") pod \"kube-auth-proxy-6bfcdd9cc7-psxhp\" (UID: \"52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07\") " pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-psxhp"
Jul 17 12:41:43.832245 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:43.832204    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:43.848402 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:43.848370    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w5hr\" (UniqueName: \"kubernetes.io/projected/52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07-kube-api-access-8w5hr\") pod \"kube-auth-proxy-6bfcdd9cc7-psxhp\" (UID: \"52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07\") " pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-psxhp"
Jul 17 12:41:43.848563 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:43.848434    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07-tls-certs\") pod \"kube-auth-proxy-6bfcdd9cc7-psxhp\" (UID: \"52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07\") " pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-psxhp"
Jul 17 12:41:43.848608 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:43.848592    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07-tmp\") pod \"kube-auth-proxy-6bfcdd9cc7-psxhp\" (UID: \"52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07\") " pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-psxhp"
Jul 17 12:41:43.850903 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:43.850878    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07-tmp\") pod \"kube-auth-proxy-6bfcdd9cc7-psxhp\" (UID: \"52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07\") " pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-psxhp"
Jul 17 12:41:43.851044 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:43.851027    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07-tls-certs\") pod \"kube-auth-proxy-6bfcdd9cc7-psxhp\" (UID: \"52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07\") " pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-psxhp"
Jul 17 12:41:43.857479 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:43.857451    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w5hr\" (UniqueName: \"kubernetes.io/projected/52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07-kube-api-access-8w5hr\") pod \"kube-auth-proxy-6bfcdd9cc7-psxhp\" (UID: \"52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07\") " pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-psxhp"
Jul 17 12:41:43.938208 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:43.938149    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:44.034301 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:44.034193    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:44.039840 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:44.039814    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-psxhp"
Jul 17 12:41:44.133616 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:44.133558    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:44.184341 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:44.184314    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bfcdd9cc7-psxhp"]
Jul 17 12:41:44.186855 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:41:44.186822    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b6c1e2_fad9_4ca4_a5b4_b38102dcdc07.slice/crio-fee42315421de25cf0acd8c8905690cb58c5fa06b5d45c00e448212572453d7e WatchSource:0}: Error finding container fee42315421de25cf0acd8c8905690cb58c5fa06b5d45c00e448212572453d7e: Status 404 returned error can't find the container with id fee42315421de25cf0acd8c8905690cb58c5fa06b5d45c00e448212572453d7e
Jul 17 12:41:44.458077 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:44.458050    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:44.683997 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:44.683950    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-r5kf4"]
Jul 17 12:41:44.687231 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:44.687208    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-r5kf4"
Jul 17 12:41:44.691719 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:44.691685    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 17 12:41:44.691719 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:44.691701    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-fk95z\""
Jul 17 12:41:44.697365 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:44.697322    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-psxhp" event={"ID":"52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07","Type":"ContainerStarted","Data":"fee42315421de25cf0acd8c8905690cb58c5fa06b5d45c00e448212572453d7e"}
Jul 17 12:41:44.698582 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:44.698556    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-r5kf4"]
Jul 17 12:41:44.757530 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:44.757428    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92d74ba2-23de-403e-9856-e54b19eb9c25-tls-certs\") pod \"model-serving-api-54d89bd5fc-r5kf4\" (UID: \"92d74ba2-23de-403e-9856-e54b19eb9c25\") " pod="opendatahub/model-serving-api-54d89bd5fc-r5kf4"
Jul 17 12:41:44.757530 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:44.757503    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6f8g\" (UniqueName: \"kubernetes.io/projected/92d74ba2-23de-403e-9856-e54b19eb9c25-kube-api-access-p6f8g\") pod \"model-serving-api-54d89bd5fc-r5kf4\" (UID: \"92d74ba2-23de-403e-9856-e54b19eb9c25\") " pod="opendatahub/model-serving-api-54d89bd5fc-r5kf4"
Jul 17 12:41:44.858654 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:44.858623    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92d74ba2-23de-403e-9856-e54b19eb9c25-tls-certs\") pod \"model-serving-api-54d89bd5fc-r5kf4\" (UID: \"92d74ba2-23de-403e-9856-e54b19eb9c25\") " pod="opendatahub/model-serving-api-54d89bd5fc-r5kf4"
Jul 17 12:41:44.858837 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:44.858673    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6f8g\" (UniqueName: \"kubernetes.io/projected/92d74ba2-23de-403e-9856-e54b19eb9c25-kube-api-access-p6f8g\") pod \"model-serving-api-54d89bd5fc-r5kf4\" (UID: \"92d74ba2-23de-403e-9856-e54b19eb9c25\") " pod="opendatahub/model-serving-api-54d89bd5fc-r5kf4"
Jul 17 12:41:44.858837 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:41:44.858788    2565 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 17 12:41:44.858909 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:41:44.858851    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92d74ba2-23de-403e-9856-e54b19eb9c25-tls-certs podName:92d74ba2-23de-403e-9856-e54b19eb9c25 nodeName:}" failed. No retries permitted until 2026-07-17 12:41:45.35883447 +0000 UTC m=+1149.490219108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/92d74ba2-23de-403e-9856-e54b19eb9c25-tls-certs") pod "model-serving-api-54d89bd5fc-r5kf4" (UID: "92d74ba2-23de-403e-9856-e54b19eb9c25") : secret "model-serving-api-tls" not found
Jul 17 12:41:44.868589 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:44.868564    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6f8g\" (UniqueName: \"kubernetes.io/projected/92d74ba2-23de-403e-9856-e54b19eb9c25-kube-api-access-p6f8g\") pod \"model-serving-api-54d89bd5fc-r5kf4\" (UID: \"92d74ba2-23de-403e-9856-e54b19eb9c25\") " pod="opendatahub/model-serving-api-54d89bd5fc-r5kf4"
Jul 17 12:41:44.913630 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:44.913589    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:45.016602 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.016523    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:45.029724 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.029675    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"]
Jul 17 12:41:45.036052 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.032311    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.036052 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.035392    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 17 12:41:45.036052 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.035733    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-54nr7\""
Jul 17 12:41:45.036052 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.035823    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 17 12:41:45.036052 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.035924    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 17 12:41:45.036052 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.035960    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 17 12:41:45.045136 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.045102    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"]
Jul 17 12:41:45.060242 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.060207    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/031d7199-245d-402c-83f4-e23ccf9e1886-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-6fvl6\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.060413 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.060250    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6n8l\" (UniqueName: \"kubernetes.io/projected/031d7199-245d-402c-83f4-e23ccf9e1886-kube-api-access-w6n8l\") pod \"istiod-openshift-gateway-7574459656-6fvl6\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.060413 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.060305    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/031d7199-245d-402c-83f4-e23ccf9e1886-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-6fvl6\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.060413 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.060403    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/031d7199-245d-402c-83f4-e23ccf9e1886-cacerts\") pod \"istiod-openshift-gateway-7574459656-6fvl6\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.060586 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.060500    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/031d7199-245d-402c-83f4-e23ccf9e1886-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-6fvl6\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.060586 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.060555    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/031d7199-245d-402c-83f4-e23ccf9e1886-local-certs\") pod \"istiod-openshift-gateway-7574459656-6fvl6\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.060586 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.060583    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/031d7199-245d-402c-83f4-e23ccf9e1886-istio-token\") pod \"istiod-openshift-gateway-7574459656-6fvl6\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.116578 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.116502    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:45.161651 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.161609    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/031d7199-245d-402c-83f4-e23ccf9e1886-local-certs\") pod \"istiod-openshift-gateway-7574459656-6fvl6\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.161818 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.161663    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/031d7199-245d-402c-83f4-e23ccf9e1886-istio-token\") pod \"istiod-openshift-gateway-7574459656-6fvl6\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.161818 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.161701    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/031d7199-245d-402c-83f4-e23ccf9e1886-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-6fvl6\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.161818 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.161729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6n8l\" (UniqueName: \"kubernetes.io/projected/031d7199-245d-402c-83f4-e23ccf9e1886-kube-api-access-w6n8l\") pod \"istiod-openshift-gateway-7574459656-6fvl6\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.161818 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.161757    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/031d7199-245d-402c-83f4-e23ccf9e1886-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-6fvl6\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.162134 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.161826    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/031d7199-245d-402c-83f4-e23ccf9e1886-cacerts\") pod \"istiod-openshift-gateway-7574459656-6fvl6\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.162134 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.161893    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/031d7199-245d-402c-83f4-e23ccf9e1886-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-6fvl6\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.162506 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.162477    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/031d7199-245d-402c-83f4-e23ccf9e1886-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-6fvl6\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.164787 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.164763    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/031d7199-245d-402c-83f4-e23ccf9e1886-local-certs\") pod \"istiod-openshift-gateway-7574459656-6fvl6\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.165004 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.164956    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/031d7199-245d-402c-83f4-e23ccf9e1886-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-6fvl6\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.165110 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.164962    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/031d7199-245d-402c-83f4-e23ccf9e1886-cacerts\") pod \"istiod-openshift-gateway-7574459656-6fvl6\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.165224 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.165203    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/031d7199-245d-402c-83f4-e23ccf9e1886-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-6fvl6\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.172954 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.172902    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/031d7199-245d-402c-83f4-e23ccf9e1886-istio-token\") pod \"istiod-openshift-gateway-7574459656-6fvl6\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.173222 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.173205    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6n8l\" (UniqueName: \"kubernetes.io/projected/031d7199-245d-402c-83f4-e23ccf9e1886-kube-api-access-w6n8l\") pod \"istiod-openshift-gateway-7574459656-6fvl6\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.348909 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.348876    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:45.363688 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.363651    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92d74ba2-23de-403e-9856-e54b19eb9c25-tls-certs\") pod \"model-serving-api-54d89bd5fc-r5kf4\" (UID: \"92d74ba2-23de-403e-9856-e54b19eb9c25\") " pod="opendatahub/model-serving-api-54d89bd5fc-r5kf4"
Jul 17 12:41:45.366775 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.366742    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92d74ba2-23de-403e-9856-e54b19eb9c25-tls-certs\") pod \"model-serving-api-54d89bd5fc-r5kf4\" (UID: \"92d74ba2-23de-403e-9856-e54b19eb9c25\") " pod="opendatahub/model-serving-api-54d89bd5fc-r5kf4"
Jul 17 12:41:45.532669 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.532639    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"]
Jul 17 12:41:45.534058 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:41:45.534027    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031d7199_245d_402c_83f4_e23ccf9e1886.slice/crio-a872a64fe67bfa66f2a4a2f923b06b41fa01183d67437d22489ee8e4f60f9d49 WatchSource:0}: Error finding container a872a64fe67bfa66f2a4a2f923b06b41fa01183d67437d22489ee8e4f60f9d49: Status 404 returned error can't find the container with id a872a64fe67bfa66f2a4a2f923b06b41fa01183d67437d22489ee8e4f60f9d49
Jul 17 12:41:45.599803 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.599713    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-r5kf4"
Jul 17 12:41:45.706148 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.706088    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6" event={"ID":"031d7199-245d-402c-83f4-e23ccf9e1886","Type":"ContainerStarted","Data":"a872a64fe67bfa66f2a4a2f923b06b41fa01183d67437d22489ee8e4f60f9d49"}
Jul 17 12:41:45.755813 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:45.755587    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-r5kf4"]
Jul 17 12:41:45.758144 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:41:45.758114    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d74ba2_23de_403e_9856_e54b19eb9c25.slice/crio-a1e5cbb97db42bf55862e562dcb35ee6686ceb6ba64bcdef4bf9f0b802e5fdb0 WatchSource:0}: Error finding container a1e5cbb97db42bf55862e562dcb35ee6686ceb6ba64bcdef4bf9f0b802e5fdb0: Status 404 returned error can't find the container with id a1e5cbb97db42bf55862e562dcb35ee6686ceb6ba64bcdef4bf9f0b802e5fdb0
Jul 17 12:41:46.025711 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:46.025620    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:46.713930 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:46.713895    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-r5kf4" event={"ID":"92d74ba2-23de-403e-9856-e54b19eb9c25","Type":"ContainerStarted","Data":"a1e5cbb97db42bf55862e562dcb35ee6686ceb6ba64bcdef4bf9f0b802e5fdb0"}
Jul 17 12:41:47.722211 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:47.722138    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-psxhp" event={"ID":"52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07","Type":"ContainerStarted","Data":"0a178cebe6bf13c573e8491d6e1558b6140459712a2f9188d25a678925503af5"}
Jul 17 12:41:47.740521 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:47.740467    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6bfcdd9cc7-psxhp" podStartSLOduration=2.17124841 podStartE2EDuration="4.74044273s" totalImagesPullingTime="2.56919432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:41:47.739735975 +0000 UTC m=+1151.871120663" watchObservedRunningTime="2026-07-17 12:41:47.74044273 +0000 UTC m=+1151.871827392"
Jul 17 12:41:48.788078 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:48.788037    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 12:41:48.788349 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:48.788121    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 12:41:49.627571 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:49.627524    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:49.732647 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:49.732608    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:49.736779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:49.736755    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6" event={"ID":"031d7199-245d-402c-83f4-e23ccf9e1886","Type":"ContainerStarted","Data":"4055cbda9ed4db5d24118d9db20aca6c2fdf2ede28dba596b79aee5474f570f5"}
Jul 17 12:41:49.736967 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:49.736947    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:49.738479 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:49.738452    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-r5kf4" event={"ID":"92d74ba2-23de-403e-9856-e54b19eb9c25","Type":"ContainerStarted","Data":"ec75197964efb2260edd2044fb2fbc698fcb2573773e666befd8e857af19a891"}
Jul 17 12:41:49.738592 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:49.738507    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-r5kf4"
Jul 17 12:41:49.738821 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:49.738802    2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-6fvl6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 12:41:49.738883 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:49.738858    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6" podUID="031d7199-245d-402c-83f4-e23ccf9e1886" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 12:41:49.772761 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:49.772698    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6" podStartSLOduration=1.5212422380000001 podStartE2EDuration="4.772680646s" totalImagesPullingTime="3.251438408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:41:49.769712366 +0000 UTC m=+1153.901097031" watchObservedRunningTime="2026-07-17 12:41:49.772680646 +0000 UTC m=+1153.904065284"
Jul 17 12:41:49.797204 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:49.797150    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-r5kf4" podStartSLOduration=2.77367685 podStartE2EDuration="5.797131144s" totalImagesPullingTime="3.023454294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:41:49.795757589 +0000 UTC m=+1153.927142251" watchObservedRunningTime="2026-07-17 12:41:49.797131144 +0000 UTC m=+1153.928515805"
Jul 17 12:41:50.350254 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:50.350211    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:50.678811 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:50.678694    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:50.744436 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:50.744412    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:51.023378 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:51.023288    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:54.100871 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.100833    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:54.195302 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.195264    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-prsfj"]
Jul 17 12:41:54.198483 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.198463    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-prsfj"
Jul 17 12:41:54.205822 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.205788    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 17 12:41:54.207130 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.207107    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-kq5c4\""
Jul 17 12:41:54.224420 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.224379    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wv999"]
Jul 17 12:41:54.226895 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.226871    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wv999"
Jul 17 12:41:54.231221 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.231194    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-cxvgq\""
Jul 17 12:41:54.231392 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.231356    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 17 12:41:54.231999 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.231955    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-prsfj"]
Jul 17 12:41:54.237387 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.237344    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:54.246831 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.246800    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wv999"]
Jul 17 12:41:54.249390 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.249356    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fa55000-c5fd-43ea-b4f6-f9759d917db7-cert\") pod \"kserve-controller-manager-d6f4b65b8-prsfj\" (UID: \"2fa55000-c5fd-43ea-b4f6-f9759d917db7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-prsfj"
Jul 17 12:41:54.249519 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.249413    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ppc2\" (UniqueName: \"kubernetes.io/projected/2fa55000-c5fd-43ea-b4f6-f9759d917db7-kube-api-access-5ppc2\") pod \"kserve-controller-manager-d6f4b65b8-prsfj\" (UID: \"2fa55000-c5fd-43ea-b4f6-f9759d917db7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-prsfj"
Jul 17 12:41:54.350747 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.350699    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb1aed0b-a75b-4d94-a525-d267307d5bce-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wv999\" (UID: \"cb1aed0b-a75b-4d94-a525-d267307d5bce\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wv999"
Jul 17 12:41:54.351011 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.350790    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ttkr\" (UniqueName: \"kubernetes.io/projected/cb1aed0b-a75b-4d94-a525-d267307d5bce-kube-api-access-2ttkr\") pod \"llmisvc-controller-manager-777c6bd7d5-wv999\" (UID: \"cb1aed0b-a75b-4d94-a525-d267307d5bce\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wv999"
Jul 17 12:41:54.351011 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.350858    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fa55000-c5fd-43ea-b4f6-f9759d917db7-cert\") pod \"kserve-controller-manager-d6f4b65b8-prsfj\" (UID: \"2fa55000-c5fd-43ea-b4f6-f9759d917db7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-prsfj"
Jul 17 12:41:54.351011 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.350905    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ppc2\" (UniqueName: \"kubernetes.io/projected/2fa55000-c5fd-43ea-b4f6-f9759d917db7-kube-api-access-5ppc2\") pod \"kserve-controller-manager-d6f4b65b8-prsfj\" (UID: \"2fa55000-c5fd-43ea-b4f6-f9759d917db7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-prsfj"
Jul 17 12:41:54.351223 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:41:54.351057    2565 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 17 12:41:54.351223 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:41:54.351158    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fa55000-c5fd-43ea-b4f6-f9759d917db7-cert podName:2fa55000-c5fd-43ea-b4f6-f9759d917db7 nodeName:}" failed. No retries permitted until 2026-07-17 12:41:54.851133252 +0000 UTC m=+1158.982517910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2fa55000-c5fd-43ea-b4f6-f9759d917db7-cert") pod "kserve-controller-manager-d6f4b65b8-prsfj" (UID: "2fa55000-c5fd-43ea-b4f6-f9759d917db7") : secret "kserve-webhook-server-cert" not found
Jul 17 12:41:54.371103 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.371076    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ppc2\" (UniqueName: \"kubernetes.io/projected/2fa55000-c5fd-43ea-b4f6-f9759d917db7-kube-api-access-5ppc2\") pod \"kserve-controller-manager-d6f4b65b8-prsfj\" (UID: \"2fa55000-c5fd-43ea-b4f6-f9759d917db7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-prsfj"
Jul 17 12:41:54.447646 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.447609    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:54.452138 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.452110    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb1aed0b-a75b-4d94-a525-d267307d5bce-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wv999\" (UID: \"cb1aed0b-a75b-4d94-a525-d267307d5bce\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wv999"
Jul 17 12:41:54.452280 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.452157    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ttkr\" (UniqueName: \"kubernetes.io/projected/cb1aed0b-a75b-4d94-a525-d267307d5bce-kube-api-access-2ttkr\") pod \"llmisvc-controller-manager-777c6bd7d5-wv999\" (UID: \"cb1aed0b-a75b-4d94-a525-d267307d5bce\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wv999"
Jul 17 12:41:54.452280 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:41:54.452268    2565 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 17 12:41:54.452344 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:41:54.452326    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb1aed0b-a75b-4d94-a525-d267307d5bce-cert podName:cb1aed0b-a75b-4d94-a525-d267307d5bce nodeName:}" failed. No retries permitted until 2026-07-17 12:41:54.952308744 +0000 UTC m=+1159.083693383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/cb1aed0b-a75b-4d94-a525-d267307d5bce-cert") pod "llmisvc-controller-manager-777c6bd7d5-wv999" (UID: "cb1aed0b-a75b-4d94-a525-d267307d5bce") : secret "llmisvc-webhook-server-cert" not found
Jul 17 12:41:54.464226 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.464187    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ttkr\" (UniqueName: \"kubernetes.io/projected/cb1aed0b-a75b-4d94-a525-d267307d5bce-kube-api-access-2ttkr\") pod \"llmisvc-controller-manager-777c6bd7d5-wv999\" (UID: \"cb1aed0b-a75b-4d94-a525-d267307d5bce\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wv999"
Jul 17 12:41:54.558128 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.558088    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:54.855271 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.855235    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fa55000-c5fd-43ea-b4f6-f9759d917db7-cert\") pod \"kserve-controller-manager-d6f4b65b8-prsfj\" (UID: \"2fa55000-c5fd-43ea-b4f6-f9759d917db7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-prsfj"
Jul 17 12:41:54.858017 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.857967    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fa55000-c5fd-43ea-b4f6-f9759d917db7-cert\") pod \"kserve-controller-manager-d6f4b65b8-prsfj\" (UID: \"2fa55000-c5fd-43ea-b4f6-f9759d917db7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-prsfj"
Jul 17 12:41:54.955892 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.955853    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb1aed0b-a75b-4d94-a525-d267307d5bce-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wv999\" (UID: \"cb1aed0b-a75b-4d94-a525-d267307d5bce\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wv999"
Jul 17 12:41:54.958561 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:54.958526    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb1aed0b-a75b-4d94-a525-d267307d5bce-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wv999\" (UID: \"cb1aed0b-a75b-4d94-a525-d267307d5bce\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wv999"
Jul 17 12:41:55.108819 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:55.108718    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-prsfj"
Jul 17 12:41:55.139109 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:55.139076    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wv999"
Jul 17 12:41:55.271090 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:55.271055    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-prsfj"]
Jul 17 12:41:55.273487 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:41:55.273460    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa55000_c5fd_43ea_b4f6_f9759d917db7.slice/crio-d0621d412e6ad855eb329e0d755e6b253758e4c3fa7596f9d67a7372867198eb WatchSource:0}: Error finding container d0621d412e6ad855eb329e0d755e6b253758e4c3fa7596f9d67a7372867198eb: Status 404 returned error can't find the container with id d0621d412e6ad855eb329e0d755e6b253758e4c3fa7596f9d67a7372867198eb
Jul 17 12:41:55.308295 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:55.308255    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wv999"]
Jul 17 12:41:55.312963 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:41:55.312937    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcb1aed0b_a75b_4d94_a525_d267307d5bce.slice/crio-1f14b7425ac94b604bb4cfb21cd172d4eaf18729d22e6e3e47a3dacf47a866b1 WatchSource:0}: Error finding container 1f14b7425ac94b604bb4cfb21cd172d4eaf18729d22e6e3e47a3dacf47a866b1: Status 404 returned error can't find the container with id 1f14b7425ac94b604bb4cfb21cd172d4eaf18729d22e6e3e47a3dacf47a866b1
Jul 17 12:41:55.407397 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:55.407301    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:55.513199 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:55.513162    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:55.609789 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:55.609755    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:41:55.771519 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:55.771422    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wv999" event={"ID":"cb1aed0b-a75b-4d94-a525-d267307d5bce","Type":"ContainerStarted","Data":"1f14b7425ac94b604bb4cfb21cd172d4eaf18729d22e6e3e47a3dacf47a866b1"}
Jul 17 12:41:55.772676 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:55.772642    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-prsfj" event={"ID":"2fa55000-c5fd-43ea-b4f6-f9759d917db7","Type":"ContainerStarted","Data":"d0621d412e6ad855eb329e0d755e6b253758e4c3fa7596f9d67a7372867198eb"}
Jul 17 12:41:57.141878 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:57.141469    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"]
Jul 17 12:41:57.141878 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:57.141706    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6" podUID="031d7199-245d-402c-83f4-e23ccf9e1886" containerName="discovery" containerID="cri-o://4055cbda9ed4db5d24118d9db20aca6c2fdf2ede28dba596b79aee5474f570f5" gracePeriod=30
Jul 17 12:41:57.787206 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:57.787165    2565 generic.go:356] "Generic (PLEG): container finished" podID="031d7199-245d-402c-83f4-e23ccf9e1886" containerID="4055cbda9ed4db5d24118d9db20aca6c2fdf2ede28dba596b79aee5474f570f5" exitCode=0
Jul 17 12:41:57.787402 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:57.787231    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6" event={"ID":"031d7199-245d-402c-83f4-e23ccf9e1886","Type":"ContainerDied","Data":"4055cbda9ed4db5d24118d9db20aca6c2fdf2ede28dba596b79aee5474f570f5"}
Jul 17 12:41:57.910046 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:57.910019    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:57.982735 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:57.982708    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/031d7199-245d-402c-83f4-e23ccf9e1886-local-certs\") pod \"031d7199-245d-402c-83f4-e23ccf9e1886\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") "
Jul 17 12:41:57.982872 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:57.982752    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/031d7199-245d-402c-83f4-e23ccf9e1886-istio-token\") pod \"031d7199-245d-402c-83f4-e23ccf9e1886\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") "
Jul 17 12:41:57.982872 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:57.982791    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/031d7199-245d-402c-83f4-e23ccf9e1886-istio-csr-ca-configmap\") pod \"031d7199-245d-402c-83f4-e23ccf9e1886\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") "
Jul 17 12:41:57.982872 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:57.982820    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/031d7199-245d-402c-83f4-e23ccf9e1886-istio-kubeconfig\") pod \"031d7199-245d-402c-83f4-e23ccf9e1886\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") "
Jul 17 12:41:57.982872 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:57.982848    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/031d7199-245d-402c-83f4-e23ccf9e1886-cacerts\") pod \"031d7199-245d-402c-83f4-e23ccf9e1886\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") "
Jul 17 12:41:57.983118 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:57.982907    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/031d7199-245d-402c-83f4-e23ccf9e1886-istio-csr-dns-cert\") pod \"031d7199-245d-402c-83f4-e23ccf9e1886\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") "
Jul 17 12:41:57.983118 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:57.982944    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6n8l\" (UniqueName: \"kubernetes.io/projected/031d7199-245d-402c-83f4-e23ccf9e1886-kube-api-access-w6n8l\") pod \"031d7199-245d-402c-83f4-e23ccf9e1886\" (UID: \"031d7199-245d-402c-83f4-e23ccf9e1886\") "
Jul 17 12:41:57.983241 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:57.983214    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/031d7199-245d-402c-83f4-e23ccf9e1886-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "031d7199-245d-402c-83f4-e23ccf9e1886" (UID: "031d7199-245d-402c-83f4-e23ccf9e1886"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:41:57.986478 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:57.986289    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/031d7199-245d-402c-83f4-e23ccf9e1886-cacerts" (OuterVolumeSpecName: "cacerts") pod "031d7199-245d-402c-83f4-e23ccf9e1886" (UID: "031d7199-245d-402c-83f4-e23ccf9e1886"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:41:57.986478 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:57.986342    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/031d7199-245d-402c-83f4-e23ccf9e1886-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "031d7199-245d-402c-83f4-e23ccf9e1886" (UID: "031d7199-245d-402c-83f4-e23ccf9e1886"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:41:57.986478 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:57.986434    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031d7199-245d-402c-83f4-e23ccf9e1886-kube-api-access-w6n8l" (OuterVolumeSpecName: "kube-api-access-w6n8l") pod "031d7199-245d-402c-83f4-e23ccf9e1886" (UID: "031d7199-245d-402c-83f4-e23ccf9e1886"). InnerVolumeSpecName "kube-api-access-w6n8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:41:57.986478 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:57.986472    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031d7199-245d-402c-83f4-e23ccf9e1886-istio-token" (OuterVolumeSpecName: "istio-token") pod "031d7199-245d-402c-83f4-e23ccf9e1886" (UID: "031d7199-245d-402c-83f4-e23ccf9e1886"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:41:57.986783 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:57.986536    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031d7199-245d-402c-83f4-e23ccf9e1886-local-certs" (OuterVolumeSpecName: "local-certs") pod "031d7199-245d-402c-83f4-e23ccf9e1886" (UID: "031d7199-245d-402c-83f4-e23ccf9e1886"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:41:57.987165 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:57.987141    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/031d7199-245d-402c-83f4-e23ccf9e1886-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "031d7199-245d-402c-83f4-e23ccf9e1886" (UID: "031d7199-245d-402c-83f4-e23ccf9e1886"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:41:58.084635 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:58.084580    2565 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/031d7199-245d-402c-83f4-e23ccf9e1886-istio-kubeconfig\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:41:58.084635 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:58.084625    2565 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/031d7199-245d-402c-83f4-e23ccf9e1886-cacerts\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:41:58.084635 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:58.084637    2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/031d7199-245d-402c-83f4-e23ccf9e1886-istio-csr-dns-cert\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:41:58.084939 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:58.084650    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6n8l\" (UniqueName: \"kubernetes.io/projected/031d7199-245d-402c-83f4-e23ccf9e1886-kube-api-access-w6n8l\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:41:58.084939 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:58.084665    2565 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/031d7199-245d-402c-83f4-e23ccf9e1886-local-certs\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:41:58.084939 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:58.084677    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/031d7199-245d-402c-83f4-e23ccf9e1886-istio-token\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:41:58.084939 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:58.084685    2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/031d7199-245d-402c-83f4-e23ccf9e1886-istio-csr-ca-configmap\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:41:58.793431 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:58.793339    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"
Jul 17 12:41:58.793431 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:58.793364    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6" event={"ID":"031d7199-245d-402c-83f4-e23ccf9e1886","Type":"ContainerDied","Data":"a872a64fe67bfa66f2a4a2f923b06b41fa01183d67437d22489ee8e4f60f9d49"}
Jul 17 12:41:58.793431 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:58.793428    2565 scope.go:117] "RemoveContainer" containerID="4055cbda9ed4db5d24118d9db20aca6c2fdf2ede28dba596b79aee5474f570f5"
Jul 17 12:41:58.794973 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:58.794947    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-prsfj" event={"ID":"2fa55000-c5fd-43ea-b4f6-f9759d917db7","Type":"ContainerStarted","Data":"e71814ac5ef7b48eb85162db5a76d234311b1a85b0148809e5a06a1491f00f1d"}
Jul 17 12:41:58.795151 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:58.795135    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-prsfj"
Jul 17 12:41:58.813417 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:58.813376    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"]
Jul 17 12:41:58.819457 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:58.819429    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-6fvl6"]
Jul 17 12:41:58.835601 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:41:58.835552    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-prsfj" podStartSLOduration=2.187779278 podStartE2EDuration="4.83553663s" totalImagesPullingTime="2.647757352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:41:58.834623808 +0000 UTC m=+1162.966008469" watchObservedRunningTime="2026-07-17 12:41:58.83553663 +0000 UTC m=+1162.966921292"
Jul 17 12:42:00.403320 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:00.403283    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="031d7199-245d-402c-83f4-e23ccf9e1886" path="/var/lib/kubelet/pods/031d7199-245d-402c-83f4-e23ccf9e1886/volumes"
Jul 17 12:42:00.748243 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:00.748157    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-r5kf4"
Jul 17 12:42:00.811159 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:00.811121    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wv999" event={"ID":"cb1aed0b-a75b-4d94-a525-d267307d5bce","Type":"ContainerStarted","Data":"f30379a672235a31d355c2611ce6c146385d2030e5855381334e2101ede035de"}
Jul 17 12:42:00.811370 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:00.811249    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wv999"
Jul 17 12:42:00.828082 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:00.828033    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wv999" podStartSLOduration=1.700170048 podStartE2EDuration="6.828015548s" totalImagesPullingTime="5.1278455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:42:00.826268702 +0000 UTC m=+1164.957653366" watchObservedRunningTime="2026-07-17 12:42:00.828015548 +0000 UTC m=+1164.959400208"
Jul 17 12:42:29.808681 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:29.808653    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-prsfj"
Jul 17 12:42:31.818925 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:31.818892    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wv999"
Jul 17 12:42:36.370877 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:36.370856    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 12:42:36.371975 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:36.371955    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 12:42:36.375703 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:36.375684    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 12:42:36.376450 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:36.376434    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 12:42:38.277525 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:38.277487    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:42:44.846563 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:44.846511    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:42:44.945858 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:44.945820    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:42:45.011737 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.011704    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nhtf9"]
Jul 17 12:42:45.012179 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.012096    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="031d7199-245d-402c-83f4-e23ccf9e1886" containerName="discovery"
Jul 17 12:42:45.012179 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.012146    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="031d7199-245d-402c-83f4-e23ccf9e1886" containerName="discovery"
Jul 17 12:42:45.012377 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.012254    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="031d7199-245d-402c-83f4-e23ccf9e1886" containerName="discovery"
Jul 17 12:42:45.018209 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.018186    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-nhtf9"
Jul 17 12:42:45.021099 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.021063    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 12:42:45.021247 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.021073    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 12:42:45.022519 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.022482    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-2vcrg\""
Jul 17 12:42:45.025172 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.025146    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nhtf9"]
Jul 17 12:42:45.027252 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.027224    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:42:45.076375 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.076341    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tp9l\" (UniqueName: \"kubernetes.io/projected/87d26f3d-ed1a-4245-9e5e-4ec44c551514-kube-api-access-2tp9l\") pod \"kuadrant-operator-catalog-nhtf9\" (UID: \"87d26f3d-ed1a-4245-9e5e-4ec44c551514\") " pod="kuadrant-system/kuadrant-operator-catalog-nhtf9"
Jul 17 12:42:45.176868 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.176779    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tp9l\" (UniqueName: \"kubernetes.io/projected/87d26f3d-ed1a-4245-9e5e-4ec44c551514-kube-api-access-2tp9l\") pod \"kuadrant-operator-catalog-nhtf9\" (UID: \"87d26f3d-ed1a-4245-9e5e-4ec44c551514\") " pod="kuadrant-system/kuadrant-operator-catalog-nhtf9"
Jul 17 12:42:45.185675 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.185645    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tp9l\" (UniqueName: \"kubernetes.io/projected/87d26f3d-ed1a-4245-9e5e-4ec44c551514-kube-api-access-2tp9l\") pod \"kuadrant-operator-catalog-nhtf9\" (UID: \"87d26f3d-ed1a-4245-9e5e-4ec44c551514\") " pod="kuadrant-system/kuadrant-operator-catalog-nhtf9"
Jul 17 12:42:45.331569 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.331530    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-nhtf9"
Jul 17 12:42:45.373979 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.373945    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nhtf9"]
Jul 17 12:42:45.379048 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.379016    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:42:45.470293 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.470268    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nhtf9"]
Jul 17 12:42:45.472634 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:42:45.472591    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d26f3d_ed1a_4245_9e5e_4ec44c551514.slice/crio-d4c0410679928e309551e535961cc1d0eebec5d49813e6f2bada06b2115f289e WatchSource:0}: Error finding container d4c0410679928e309551e535961cc1d0eebec5d49813e6f2bada06b2115f289e: Status 404 returned error can't find the container with id d4c0410679928e309551e535961cc1d0eebec5d49813e6f2bada06b2115f289e
Jul 17 12:42:45.583832 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.583793    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ckqdg"]
Jul 17 12:42:45.589170 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.589140    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ckqdg"
Jul 17 12:42:45.596196 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.596171    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ckqdg"]
Jul 17 12:42:45.680457 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.680419    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbz79\" (UniqueName: \"kubernetes.io/projected/31148710-faca-46f4-af71-e367aabe39e6-kube-api-access-xbz79\") pod \"kuadrant-operator-catalog-ckqdg\" (UID: \"31148710-faca-46f4-af71-e367aabe39e6\") " pod="kuadrant-system/kuadrant-operator-catalog-ckqdg"
Jul 17 12:42:45.781771 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.781692    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbz79\" (UniqueName: \"kubernetes.io/projected/31148710-faca-46f4-af71-e367aabe39e6-kube-api-access-xbz79\") pod \"kuadrant-operator-catalog-ckqdg\" (UID: \"31148710-faca-46f4-af71-e367aabe39e6\") " pod="kuadrant-system/kuadrant-operator-catalog-ckqdg"
Jul 17 12:42:45.790758 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.790730    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbz79\" (UniqueName: \"kubernetes.io/projected/31148710-faca-46f4-af71-e367aabe39e6-kube-api-access-xbz79\") pod \"kuadrant-operator-catalog-ckqdg\" (UID: \"31148710-faca-46f4-af71-e367aabe39e6\") " pod="kuadrant-system/kuadrant-operator-catalog-ckqdg"
Jul 17 12:42:45.803580 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.803535    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:42:45.899940 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:45.899909    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ckqdg"
Jul 17 12:42:46.032880 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:46.032795    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-nhtf9" event={"ID":"87d26f3d-ed1a-4245-9e5e-4ec44c551514","Type":"ContainerStarted","Data":"d4c0410679928e309551e535961cc1d0eebec5d49813e6f2bada06b2115f289e"}
Jul 17 12:42:46.037670 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:46.037650    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ckqdg"]
Jul 17 12:42:46.039301 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:42:46.039282    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31148710_faca_46f4_af71_e367aabe39e6.slice/crio-c8c3bd40f2bde0b1e3dfc52517ada1ebe9d324bb71ead899a26ed5d923fea515 WatchSource:0}: Error finding container c8c3bd40f2bde0b1e3dfc52517ada1ebe9d324bb71ead899a26ed5d923fea515: Status 404 returned error can't find the container with id c8c3bd40f2bde0b1e3dfc52517ada1ebe9d324bb71ead899a26ed5d923fea515
Jul 17 12:42:47.041471 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:47.041430    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ckqdg" event={"ID":"31148710-faca-46f4-af71-e367aabe39e6","Type":"ContainerStarted","Data":"c8c3bd40f2bde0b1e3dfc52517ada1ebe9d324bb71ead899a26ed5d923fea515"}
Jul 17 12:42:48.047519 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:48.047480    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ckqdg" event={"ID":"31148710-faca-46f4-af71-e367aabe39e6","Type":"ContainerStarted","Data":"a336332c533ccc9143e5848cdc479abb58485421219d2ee85278ee4e0a6571c0"}
Jul 17 12:42:48.064355 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:48.064301    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-ckqdg" podStartSLOduration=1.270196501 podStartE2EDuration="3.064285455s" totalImagesPullingTime="1.794088954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:42:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:42:48.06275273 +0000 UTC m=+1212.194137410" watchObservedRunningTime="2026-07-17 12:42:48.064285455 +0000 UTC m=+1212.195670094"
Jul 17 12:42:48.517634 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:48.517599    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:42:49.054736 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:49.054692    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-nhtf9" event={"ID":"87d26f3d-ed1a-4245-9e5e-4ec44c551514","Type":"ContainerStarted","Data":"df054b07a2b98aef7d40a68ed5d955036acc2f1d2c970081124817fd30537433"}
Jul 17 12:42:49.055262 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:49.054753    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-nhtf9" podUID="87d26f3d-ed1a-4245-9e5e-4ec44c551514" containerName="registry-server" containerID="cri-o://df054b07a2b98aef7d40a68ed5d955036acc2f1d2c970081124817fd30537433" gracePeriod=2
Jul 17 12:42:49.070601 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:49.070546    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-nhtf9" podStartSLOduration=1.753551954 podStartE2EDuration="5.070525434s" totalImagesPullingTime="3.31697348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:42:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:42:49.069455625 +0000 UTC m=+1213.200840310" watchObservedRunningTime="2026-07-17 12:42:49.070525434 +0000 UTC m=+1213.201910096"
Jul 17 12:42:49.306387 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:49.306316    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-nhtf9"
Jul 17 12:42:49.413715 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:49.413684    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tp9l\" (UniqueName: \"kubernetes.io/projected/87d26f3d-ed1a-4245-9e5e-4ec44c551514-kube-api-access-2tp9l\") pod \"87d26f3d-ed1a-4245-9e5e-4ec44c551514\" (UID: \"87d26f3d-ed1a-4245-9e5e-4ec44c551514\") "
Jul 17 12:42:49.415962 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:49.415933    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d26f3d-ed1a-4245-9e5e-4ec44c551514-kube-api-access-2tp9l" (OuterVolumeSpecName: "kube-api-access-2tp9l") pod "87d26f3d-ed1a-4245-9e5e-4ec44c551514" (UID: "87d26f3d-ed1a-4245-9e5e-4ec44c551514"). InnerVolumeSpecName "kube-api-access-2tp9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:42:49.515281 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:49.515245    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tp9l\" (UniqueName: \"kubernetes.io/projected/87d26f3d-ed1a-4245-9e5e-4ec44c551514-kube-api-access-2tp9l\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:42:50.060935 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:50.060900    2565 generic.go:356] "Generic (PLEG): container finished" podID="87d26f3d-ed1a-4245-9e5e-4ec44c551514" containerID="df054b07a2b98aef7d40a68ed5d955036acc2f1d2c970081124817fd30537433" exitCode=0
Jul 17 12:42:50.061439 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:50.060945    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-nhtf9"
Jul 17 12:42:50.061439 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:50.060955    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-nhtf9" event={"ID":"87d26f3d-ed1a-4245-9e5e-4ec44c551514","Type":"ContainerDied","Data":"df054b07a2b98aef7d40a68ed5d955036acc2f1d2c970081124817fd30537433"}
Jul 17 12:42:50.061439 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:50.061018    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-nhtf9" event={"ID":"87d26f3d-ed1a-4245-9e5e-4ec44c551514","Type":"ContainerDied","Data":"d4c0410679928e309551e535961cc1d0eebec5d49813e6f2bada06b2115f289e"}
Jul 17 12:42:50.061439 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:50.061040    2565 scope.go:117] "RemoveContainer" containerID="df054b07a2b98aef7d40a68ed5d955036acc2f1d2c970081124817fd30537433"
Jul 17 12:42:50.070458 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:50.070440    2565 scope.go:117] "RemoveContainer" containerID="df054b07a2b98aef7d40a68ed5d955036acc2f1d2c970081124817fd30537433"
Jul 17 12:42:50.070699 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:42:50.070678    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df054b07a2b98aef7d40a68ed5d955036acc2f1d2c970081124817fd30537433\": container with ID starting with df054b07a2b98aef7d40a68ed5d955036acc2f1d2c970081124817fd30537433 not found: ID does not exist" containerID="df054b07a2b98aef7d40a68ed5d955036acc2f1d2c970081124817fd30537433"
Jul 17 12:42:50.070755 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:50.070714    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df054b07a2b98aef7d40a68ed5d955036acc2f1d2c970081124817fd30537433"} err="failed to get container status \"df054b07a2b98aef7d40a68ed5d955036acc2f1d2c970081124817fd30537433\": rpc error: code = NotFound desc = could not find container \"df054b07a2b98aef7d40a68ed5d955036acc2f1d2c970081124817fd30537433\": container with ID starting with df054b07a2b98aef7d40a68ed5d955036acc2f1d2c970081124817fd30537433 not found: ID does not exist"
Jul 17 12:42:50.081695 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:50.081669    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nhtf9"]
Jul 17 12:42:50.085515 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:50.085495    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nhtf9"]
Jul 17 12:42:50.402763 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:50.402691    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87d26f3d-ed1a-4245-9e5e-4ec44c551514" path="/var/lib/kubelet/pods/87d26f3d-ed1a-4245-9e5e-4ec44c551514/volumes"
Jul 17 12:42:53.310795 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:53.310757    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:42:54.282353 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:54.282310    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:42:54.340767 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:54.340731    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:42:55.900941 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:55.900901    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-ckqdg"
Jul 17 12:42:55.901446 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:55.900970    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-ckqdg"
Jul 17 12:42:55.923674 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:55.923650    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-ckqdg"
Jul 17 12:42:56.112075 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:42:56.112046    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-ckqdg"
Jul 17 12:43:01.221351 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.221317    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg"]
Jul 17 12:43:01.221817 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.221668    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87d26f3d-ed1a-4245-9e5e-4ec44c551514" containerName="registry-server"
Jul 17 12:43:01.221817 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.221680    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="87d26f3d-ed1a-4245-9e5e-4ec44c551514" containerName="registry-server"
Jul 17 12:43:01.221817 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.221756    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="87d26f3d-ed1a-4245-9e5e-4ec44c551514" containerName="registry-server"
Jul 17 12:43:01.225456 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.225441    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg"
Jul 17 12:43:01.228021 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.228003    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mb4gp\""
Jul 17 12:43:01.233212 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.233173    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg"]
Jul 17 12:43:01.315890 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.315858    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd980ea9-8de2-47a5-ae4b-2e5b17764ff0-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg\" (UID: \"fd980ea9-8de2-47a5-ae4b-2e5b17764ff0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg"
Jul 17 12:43:01.316078 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.315893    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd980ea9-8de2-47a5-ae4b-2e5b17764ff0-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg\" (UID: \"fd980ea9-8de2-47a5-ae4b-2e5b17764ff0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg"
Jul 17 12:43:01.316078 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.315952    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5bb8\" (UniqueName: \"kubernetes.io/projected/fd980ea9-8de2-47a5-ae4b-2e5b17764ff0-kube-api-access-h5bb8\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg\" (UID: \"fd980ea9-8de2-47a5-ae4b-2e5b17764ff0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg"
Jul 17 12:43:01.417092 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.417060    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5bb8\" (UniqueName: \"kubernetes.io/projected/fd980ea9-8de2-47a5-ae4b-2e5b17764ff0-kube-api-access-h5bb8\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg\" (UID: \"fd980ea9-8de2-47a5-ae4b-2e5b17764ff0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg"
Jul 17 12:43:01.417255 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.417154    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd980ea9-8de2-47a5-ae4b-2e5b17764ff0-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg\" (UID: \"fd980ea9-8de2-47a5-ae4b-2e5b17764ff0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg"
Jul 17 12:43:01.417255 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.417175    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd980ea9-8de2-47a5-ae4b-2e5b17764ff0-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg\" (UID: \"fd980ea9-8de2-47a5-ae4b-2e5b17764ff0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg"
Jul 17 12:43:01.417594 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.417575    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd980ea9-8de2-47a5-ae4b-2e5b17764ff0-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg\" (UID: \"fd980ea9-8de2-47a5-ae4b-2e5b17764ff0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg"
Jul 17 12:43:01.417638 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.417588    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd980ea9-8de2-47a5-ae4b-2e5b17764ff0-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg\" (UID: \"fd980ea9-8de2-47a5-ae4b-2e5b17764ff0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg"
Jul 17 12:43:01.425602 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.425573    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5bb8\" (UniqueName: \"kubernetes.io/projected/fd980ea9-8de2-47a5-ae4b-2e5b17764ff0-kube-api-access-h5bb8\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg\" (UID: \"fd980ea9-8de2-47a5-ae4b-2e5b17764ff0\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg"
Jul 17 12:43:01.536273 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.536202    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg"
Jul 17 12:43:01.666380 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.666359    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg"]
Jul 17 12:43:01.668505 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:43:01.668457    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd980ea9_8de2_47a5_ae4b_2e5b17764ff0.slice/crio-5f76272432b2d2cf3a67f35976070405cfd3e3a3b906c79e9186562688a89b8e WatchSource:0}: Error finding container 5f76272432b2d2cf3a67f35976070405cfd3e3a3b906c79e9186562688a89b8e: Status 404 returned error can't find the container with id 5f76272432b2d2cf3a67f35976070405cfd3e3a3b906c79e9186562688a89b8e
Jul 17 12:43:01.817588 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.817553    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg"]
Jul 17 12:43:01.821316 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.821298    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg"
Jul 17 12:43:01.829446 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.829423    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg"]
Jul 17 12:43:01.921312 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.921281    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b80e493-a458-4093-9173-922689bce863-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg\" (UID: \"2b80e493-a458-4093-9173-922689bce863\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg"
Jul 17 12:43:01.921312 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.921323    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85cqq\" (UniqueName: \"kubernetes.io/projected/2b80e493-a458-4093-9173-922689bce863-kube-api-access-85cqq\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg\" (UID: \"2b80e493-a458-4093-9173-922689bce863\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg"
Jul 17 12:43:01.921549 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:01.921438    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b80e493-a458-4093-9173-922689bce863-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg\" (UID: \"2b80e493-a458-4093-9173-922689bce863\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg"
Jul 17 12:43:02.022487 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.022448    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b80e493-a458-4093-9173-922689bce863-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg\" (UID: \"2b80e493-a458-4093-9173-922689bce863\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg"
Jul 17 12:43:02.022696 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.022504    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b80e493-a458-4093-9173-922689bce863-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg\" (UID: \"2b80e493-a458-4093-9173-922689bce863\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg"
Jul 17 12:43:02.022696 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.022536    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85cqq\" (UniqueName: \"kubernetes.io/projected/2b80e493-a458-4093-9173-922689bce863-kube-api-access-85cqq\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg\" (UID: \"2b80e493-a458-4093-9173-922689bce863\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg"
Jul 17 12:43:02.022922 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.022899    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b80e493-a458-4093-9173-922689bce863-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg\" (UID: \"2b80e493-a458-4093-9173-922689bce863\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg"
Jul 17 12:43:02.023027 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.022933    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b80e493-a458-4093-9173-922689bce863-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg\" (UID: \"2b80e493-a458-4093-9173-922689bce863\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg"
Jul 17 12:43:02.031031 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.031013    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85cqq\" (UniqueName: \"kubernetes.io/projected/2b80e493-a458-4093-9173-922689bce863-kube-api-access-85cqq\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg\" (UID: \"2b80e493-a458-4093-9173-922689bce863\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg"
Jul 17 12:43:02.120106 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.120020    2565 generic.go:356] "Generic (PLEG): container finished" podID="fd980ea9-8de2-47a5-ae4b-2e5b17764ff0" containerID="8c814417f89776c7f33e7d804acb0da674821cfa2a5862aa0abc98a2a06f751f" exitCode=0
Jul 17 12:43:02.120106 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.120066    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg" event={"ID":"fd980ea9-8de2-47a5-ae4b-2e5b17764ff0","Type":"ContainerDied","Data":"8c814417f89776c7f33e7d804acb0da674821cfa2a5862aa0abc98a2a06f751f"}
Jul 17 12:43:02.120106 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.120100    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg" event={"ID":"fd980ea9-8de2-47a5-ae4b-2e5b17764ff0","Type":"ContainerStarted","Data":"5f76272432b2d2cf3a67f35976070405cfd3e3a3b906c79e9186562688a89b8e"}
Jul 17 12:43:02.131534 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.131511    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg"
Jul 17 12:43:02.217558 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.217524    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft"]
Jul 17 12:43:02.222003 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.221962    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft"
Jul 17 12:43:02.228829 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.228799    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft"]
Jul 17 12:43:02.279399 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.279195    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg"]
Jul 17 12:43:02.282116 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:43:02.282094    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b80e493_a458_4093_9173_922689bce863.slice/crio-b8bdf97d58891ef80c4145e933f049e6e212368ab7070cc4aee72e6c642c50b8 WatchSource:0}: Error finding container b8bdf97d58891ef80c4145e933f049e6e212368ab7070cc4aee72e6c642c50b8: Status 404 returned error can't find the container with id b8bdf97d58891ef80c4145e933f049e6e212368ab7070cc4aee72e6c642c50b8
Jul 17 12:43:02.326250 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.326216    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/488f355f-da9e-448e-b066-ed881e791765-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft\" (UID: \"488f355f-da9e-448e-b066-ed881e791765\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft"
Jul 17 12:43:02.326399 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.326323    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v776d\" (UniqueName: \"kubernetes.io/projected/488f355f-da9e-448e-b066-ed881e791765-kube-api-access-v776d\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft\" (UID: \"488f355f-da9e-448e-b066-ed881e791765\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft"
Jul 17 12:43:02.326467 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.326410    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/488f355f-da9e-448e-b066-ed881e791765-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft\" (UID: \"488f355f-da9e-448e-b066-ed881e791765\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft"
Jul 17 12:43:02.427081 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.427040    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/488f355f-da9e-448e-b066-ed881e791765-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft\" (UID: \"488f355f-da9e-448e-b066-ed881e791765\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft"
Jul 17 12:43:02.427248 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.427094    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v776d\" (UniqueName: \"kubernetes.io/projected/488f355f-da9e-448e-b066-ed881e791765-kube-api-access-v776d\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft\" (UID: \"488f355f-da9e-448e-b066-ed881e791765\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft"
Jul 17 12:43:02.427248 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.427122    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/488f355f-da9e-448e-b066-ed881e791765-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft\" (UID: \"488f355f-da9e-448e-b066-ed881e791765\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft"
Jul 17 12:43:02.427431 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.427411    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/488f355f-da9e-448e-b066-ed881e791765-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft\" (UID: \"488f355f-da9e-448e-b066-ed881e791765\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft"
Jul 17 12:43:02.427479 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.427445    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/488f355f-da9e-448e-b066-ed881e791765-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft\" (UID: \"488f355f-da9e-448e-b066-ed881e791765\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft"
Jul 17 12:43:02.436190 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.436166    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v776d\" (UniqueName: \"kubernetes.io/projected/488f355f-da9e-448e-b066-ed881e791765-kube-api-access-v776d\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft\" (UID: \"488f355f-da9e-448e-b066-ed881e791765\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft"
Jul 17 12:43:02.535055 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.535015    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft"
Jul 17 12:43:02.622745 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.622712    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp"]
Jul 17 12:43:02.628244 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.628221    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp"
Jul 17 12:43:02.634419 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.634316    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp"]
Jul 17 12:43:02.675817 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.675786    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft"]
Jul 17 12:43:02.681668 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:43:02.681636    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488f355f_da9e_448e_b066_ed881e791765.slice/crio-e6ad4a59e63e0a80fb72bf91cd8d0dce2a7f562e96e8d1e2f20089e30da49d0e WatchSource:0}: Error finding container e6ad4a59e63e0a80fb72bf91cd8d0dce2a7f562e96e8d1e2f20089e30da49d0e: Status 404 returned error can't find the container with id e6ad4a59e63e0a80fb72bf91cd8d0dce2a7f562e96e8d1e2f20089e30da49d0e
Jul 17 12:43:02.730081 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.730049    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80e4bd69-feb8-496a-966e-6cfea38936e7-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp\" (UID: \"80e4bd69-feb8-496a-966e-6cfea38936e7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp"
Jul 17 12:43:02.730228 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.730089    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80e4bd69-feb8-496a-966e-6cfea38936e7-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp\" (UID: \"80e4bd69-feb8-496a-966e-6cfea38936e7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp"
Jul 17 12:43:02.730228 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.730119    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dxlv\" (UniqueName: \"kubernetes.io/projected/80e4bd69-feb8-496a-966e-6cfea38936e7-kube-api-access-5dxlv\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp\" (UID: \"80e4bd69-feb8-496a-966e-6cfea38936e7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp"
Jul 17 12:43:02.831234 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.831196    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80e4bd69-feb8-496a-966e-6cfea38936e7-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp\" (UID: \"80e4bd69-feb8-496a-966e-6cfea38936e7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp"
Jul 17 12:43:02.831234 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.831241    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80e4bd69-feb8-496a-966e-6cfea38936e7-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp\" (UID: \"80e4bd69-feb8-496a-966e-6cfea38936e7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp"
Jul 17 12:43:02.831533 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.831272    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dxlv\" (UniqueName: \"kubernetes.io/projected/80e4bd69-feb8-496a-966e-6cfea38936e7-kube-api-access-5dxlv\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp\" (UID: \"80e4bd69-feb8-496a-966e-6cfea38936e7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp"
Jul 17 12:43:02.831677 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.831652    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80e4bd69-feb8-496a-966e-6cfea38936e7-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp\" (UID: \"80e4bd69-feb8-496a-966e-6cfea38936e7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp"
Jul 17 12:43:02.831737 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.831703    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80e4bd69-feb8-496a-966e-6cfea38936e7-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp\" (UID: \"80e4bd69-feb8-496a-966e-6cfea38936e7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp"
Jul 17 12:43:02.840878 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.840854    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dxlv\" (UniqueName: \"kubernetes.io/projected/80e4bd69-feb8-496a-966e-6cfea38936e7-kube-api-access-5dxlv\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp\" (UID: \"80e4bd69-feb8-496a-966e-6cfea38936e7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp"
Jul 17 12:43:02.940441 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:02.940356    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp"
Jul 17 12:43:03.076456 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:03.076431    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp"]
Jul 17 12:43:03.077814 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:43:03.077786    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e4bd69_feb8_496a_966e_6cfea38936e7.slice/crio-186f8f68c26f96c84a2ab4f20d0c2a8eb7e10384338a446ca3d028d08c44c3e3 WatchSource:0}: Error finding container 186f8f68c26f96c84a2ab4f20d0c2a8eb7e10384338a446ca3d028d08c44c3e3: Status 404 returned error can't find the container with id 186f8f68c26f96c84a2ab4f20d0c2a8eb7e10384338a446ca3d028d08c44c3e3
Jul 17 12:43:03.136698 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:03.136662    2565 generic.go:356] "Generic (PLEG): container finished" podID="488f355f-da9e-448e-b066-ed881e791765" containerID="e2557a01c78dc58c146fff26d2643ba14f1e8843401ce2fcedba272c93afb308" exitCode=0
Jul 17 12:43:03.136887 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:03.136757    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft" event={"ID":"488f355f-da9e-448e-b066-ed881e791765","Type":"ContainerDied","Data":"e2557a01c78dc58c146fff26d2643ba14f1e8843401ce2fcedba272c93afb308"}
Jul 17 12:43:03.136887 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:03.136822    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft" event={"ID":"488f355f-da9e-448e-b066-ed881e791765","Type":"ContainerStarted","Data":"e6ad4a59e63e0a80fb72bf91cd8d0dce2a7f562e96e8d1e2f20089e30da49d0e"}
Jul 17 12:43:03.138846 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:03.138822    2565 generic.go:356] "Generic (PLEG): container finished" podID="2b80e493-a458-4093-9173-922689bce863" containerID="aeb57f3793ccebffd6b8f922deb1e187100e1e1ed36667878bf2d238bf4a43ef" exitCode=0
Jul 17 12:43:03.138965 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:03.138905    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg" event={"ID":"2b80e493-a458-4093-9173-922689bce863","Type":"ContainerDied","Data":"aeb57f3793ccebffd6b8f922deb1e187100e1e1ed36667878bf2d238bf4a43ef"}
Jul 17 12:43:03.138965 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:03.138960    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg" event={"ID":"2b80e493-a458-4093-9173-922689bce863","Type":"ContainerStarted","Data":"b8bdf97d58891ef80c4145e933f049e6e212368ab7070cc4aee72e6c642c50b8"}
Jul 17 12:43:03.140838 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:03.140811    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp" event={"ID":"80e4bd69-feb8-496a-966e-6cfea38936e7","Type":"ContainerStarted","Data":"186f8f68c26f96c84a2ab4f20d0c2a8eb7e10384338a446ca3d028d08c44c3e3"}
Jul 17 12:43:04.147722 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:04.147635    2565 generic.go:356] "Generic (PLEG): container finished" podID="2b80e493-a458-4093-9173-922689bce863" containerID="de84f257711c00b4f6518240f4a11da3cd8f03c33eacc7a0bd81f04a720154a5" exitCode=0
Jul 17 12:43:04.148164 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:04.147735    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg" event={"ID":"2b80e493-a458-4093-9173-922689bce863","Type":"ContainerDied","Data":"de84f257711c00b4f6518240f4a11da3cd8f03c33eacc7a0bd81f04a720154a5"}
Jul 17 12:43:04.149431 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:04.149407    2565 generic.go:356] "Generic (PLEG): container finished" podID="80e4bd69-feb8-496a-966e-6cfea38936e7" containerID="cad83dc66802ef15c95e1b0a6fd6ade8c9e89da121f4855489f703786a6a107d" exitCode=0
Jul 17 12:43:04.149491 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:04.149432    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp" event={"ID":"80e4bd69-feb8-496a-966e-6cfea38936e7","Type":"ContainerDied","Data":"cad83dc66802ef15c95e1b0a6fd6ade8c9e89da121f4855489f703786a6a107d"}
Jul 17 12:43:05.158051 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:05.158017    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg" event={"ID":"2b80e493-a458-4093-9173-922689bce863","Type":"ContainerDied","Data":"df7cac32069a8ff64437370b4d181616fe6abf060455c68c6608b8cec5dd5141"}
Jul 17 12:43:05.158051 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:05.157971    2565 generic.go:356] "Generic (PLEG): container finished" podID="2b80e493-a458-4093-9173-922689bce863" containerID="df7cac32069a8ff64437370b4d181616fe6abf060455c68c6608b8cec5dd5141" exitCode=0
Jul 17 12:43:05.510005 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:05.509897    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:05.610749 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:05.610710    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:05.974456 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:05.974423    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:06.166304 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:06.166216    2565 generic.go:356] "Generic (PLEG): container finished" podID="fd980ea9-8de2-47a5-ae4b-2e5b17764ff0" containerID="e8d896bbb5536a6b96dcd96458747e32d42d9543b081120df21126ef5e7a8144" exitCode=0
Jul 17 12:43:06.166304 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:06.166292    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg" event={"ID":"fd980ea9-8de2-47a5-ae4b-2e5b17764ff0","Type":"ContainerDied","Data":"e8d896bbb5536a6b96dcd96458747e32d42d9543b081120df21126ef5e7a8144"}
Jul 17 12:43:06.168307 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:06.168285    2565 generic.go:356] "Generic (PLEG): container finished" podID="80e4bd69-feb8-496a-966e-6cfea38936e7" containerID="d193371cb7952f297e7ec5c0bbb3a24a9843488ddbdfa2e5f2019682d6070557" exitCode=0
Jul 17 12:43:06.168417 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:06.168365    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp" event={"ID":"80e4bd69-feb8-496a-966e-6cfea38936e7","Type":"ContainerDied","Data":"d193371cb7952f297e7ec5c0bbb3a24a9843488ddbdfa2e5f2019682d6070557"}
Jul 17 12:43:06.318382 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:06.318348    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg"
Jul 17 12:43:06.466038 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:06.465941    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85cqq\" (UniqueName: \"kubernetes.io/projected/2b80e493-a458-4093-9173-922689bce863-kube-api-access-85cqq\") pod \"2b80e493-a458-4093-9173-922689bce863\" (UID: \"2b80e493-a458-4093-9173-922689bce863\") "
Jul 17 12:43:06.466038 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:06.466031    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b80e493-a458-4093-9173-922689bce863-bundle\") pod \"2b80e493-a458-4093-9173-922689bce863\" (UID: \"2b80e493-a458-4093-9173-922689bce863\") "
Jul 17 12:43:06.466220 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:06.466054    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b80e493-a458-4093-9173-922689bce863-util\") pod \"2b80e493-a458-4093-9173-922689bce863\" (UID: \"2b80e493-a458-4093-9173-922689bce863\") "
Jul 17 12:43:06.466558 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:06.466534    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b80e493-a458-4093-9173-922689bce863-bundle" (OuterVolumeSpecName: "bundle") pod "2b80e493-a458-4093-9173-922689bce863" (UID: "2b80e493-a458-4093-9173-922689bce863"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:43:06.468254 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:06.468228    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b80e493-a458-4093-9173-922689bce863-kube-api-access-85cqq" (OuterVolumeSpecName: "kube-api-access-85cqq") pod "2b80e493-a458-4093-9173-922689bce863" (UID: "2b80e493-a458-4093-9173-922689bce863"). InnerVolumeSpecName "kube-api-access-85cqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:43:06.470945 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:06.470911    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b80e493-a458-4093-9173-922689bce863-util" (OuterVolumeSpecName: "util") pod "2b80e493-a458-4093-9173-922689bce863" (UID: "2b80e493-a458-4093-9173-922689bce863"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:43:06.567366 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:06.567315    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85cqq\" (UniqueName: \"kubernetes.io/projected/2b80e493-a458-4093-9173-922689bce863-kube-api-access-85cqq\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:43:06.567366 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:06.567359    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b80e493-a458-4093-9173-922689bce863-bundle\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:43:06.567366 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:06.567370    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b80e493-a458-4093-9173-922689bce863-util\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:43:07.175504 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:07.175473    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg"
Jul 17 12:43:07.175504 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:07.175482    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg" event={"ID":"2b80e493-a458-4093-9173-922689bce863","Type":"ContainerDied","Data":"b8bdf97d58891ef80c4145e933f049e6e212368ab7070cc4aee72e6c642c50b8"}
Jul 17 12:43:07.176082 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:07.175565    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8bdf97d58891ef80c4145e933f049e6e212368ab7070cc4aee72e6c642c50b8"
Jul 17 12:43:07.177424 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:07.177398    2565 generic.go:356] "Generic (PLEG): container finished" podID="80e4bd69-feb8-496a-966e-6cfea38936e7" containerID="a00d3aaa714733d0cf016ebe893acf3ddb13d39b11b4e22cd00786a0a4e88220" exitCode=0
Jul 17 12:43:07.177538 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:07.177472    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp" event={"ID":"80e4bd69-feb8-496a-966e-6cfea38936e7","Type":"ContainerDied","Data":"a00d3aaa714733d0cf016ebe893acf3ddb13d39b11b4e22cd00786a0a4e88220"}
Jul 17 12:43:07.179540 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:07.179516    2565 generic.go:356] "Generic (PLEG): container finished" podID="fd980ea9-8de2-47a5-ae4b-2e5b17764ff0" containerID="3f16565aefcf086fbf313a2b8b402fe45c3cf811fcea81408567cde0c9b69165" exitCode=0
Jul 17 12:43:07.179650 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:07.179565    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg" event={"ID":"fd980ea9-8de2-47a5-ae4b-2e5b17764ff0","Type":"ContainerDied","Data":"3f16565aefcf086fbf313a2b8b402fe45c3cf811fcea81408567cde0c9b69165"}
Jul 17 12:43:08.322822 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:08.322795    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp"
Jul 17 12:43:08.360612 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:08.360583    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg"
Jul 17 12:43:08.487826 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:08.487734    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dxlv\" (UniqueName: \"kubernetes.io/projected/80e4bd69-feb8-496a-966e-6cfea38936e7-kube-api-access-5dxlv\") pod \"80e4bd69-feb8-496a-966e-6cfea38936e7\" (UID: \"80e4bd69-feb8-496a-966e-6cfea38936e7\") "
Jul 17 12:43:08.487826 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:08.487775    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80e4bd69-feb8-496a-966e-6cfea38936e7-bundle\") pod \"80e4bd69-feb8-496a-966e-6cfea38936e7\" (UID: \"80e4bd69-feb8-496a-966e-6cfea38936e7\") "
Jul 17 12:43:08.487826 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:08.487831    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd980ea9-8de2-47a5-ae4b-2e5b17764ff0-bundle\") pod \"fd980ea9-8de2-47a5-ae4b-2e5b17764ff0\" (UID: \"fd980ea9-8de2-47a5-ae4b-2e5b17764ff0\") "
Jul 17 12:43:08.488136 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:08.487863    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5bb8\" (UniqueName: \"kubernetes.io/projected/fd980ea9-8de2-47a5-ae4b-2e5b17764ff0-kube-api-access-h5bb8\") pod \"fd980ea9-8de2-47a5-ae4b-2e5b17764ff0\" (UID: \"fd980ea9-8de2-47a5-ae4b-2e5b17764ff0\") "
Jul 17 12:43:08.488136 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:08.487909    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd980ea9-8de2-47a5-ae4b-2e5b17764ff0-util\") pod \"fd980ea9-8de2-47a5-ae4b-2e5b17764ff0\" (UID: \"fd980ea9-8de2-47a5-ae4b-2e5b17764ff0\") "
Jul 17 12:43:08.488136 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:08.487956    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80e4bd69-feb8-496a-966e-6cfea38936e7-util\") pod \"80e4bd69-feb8-496a-966e-6cfea38936e7\" (UID: \"80e4bd69-feb8-496a-966e-6cfea38936e7\") "
Jul 17 12:43:08.488492 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:08.488464    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd980ea9-8de2-47a5-ae4b-2e5b17764ff0-bundle" (OuterVolumeSpecName: "bundle") pod "fd980ea9-8de2-47a5-ae4b-2e5b17764ff0" (UID: "fd980ea9-8de2-47a5-ae4b-2e5b17764ff0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:43:08.488768 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:08.488614    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e4bd69-feb8-496a-966e-6cfea38936e7-bundle" (OuterVolumeSpecName: "bundle") pod "80e4bd69-feb8-496a-966e-6cfea38936e7" (UID: "80e4bd69-feb8-496a-966e-6cfea38936e7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:43:08.490275 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:08.490249    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd980ea9-8de2-47a5-ae4b-2e5b17764ff0-kube-api-access-h5bb8" (OuterVolumeSpecName: "kube-api-access-h5bb8") pod "fd980ea9-8de2-47a5-ae4b-2e5b17764ff0" (UID: "fd980ea9-8de2-47a5-ae4b-2e5b17764ff0"). InnerVolumeSpecName "kube-api-access-h5bb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:43:08.490472 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:08.490451    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e4bd69-feb8-496a-966e-6cfea38936e7-kube-api-access-5dxlv" (OuterVolumeSpecName: "kube-api-access-5dxlv") pod "80e4bd69-feb8-496a-966e-6cfea38936e7" (UID: "80e4bd69-feb8-496a-966e-6cfea38936e7"). InnerVolumeSpecName "kube-api-access-5dxlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:43:08.493464 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:08.493440    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd980ea9-8de2-47a5-ae4b-2e5b17764ff0-util" (OuterVolumeSpecName: "util") pod "fd980ea9-8de2-47a5-ae4b-2e5b17764ff0" (UID: "fd980ea9-8de2-47a5-ae4b-2e5b17764ff0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:43:08.493621 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:08.493598    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e4bd69-feb8-496a-966e-6cfea38936e7-util" (OuterVolumeSpecName: "util") pod "80e4bd69-feb8-496a-966e-6cfea38936e7" (UID: "80e4bd69-feb8-496a-966e-6cfea38936e7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:43:08.589274 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:08.589236    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80e4bd69-feb8-496a-966e-6cfea38936e7-util\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:43:08.589274 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:08.589268    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dxlv\" (UniqueName: \"kubernetes.io/projected/80e4bd69-feb8-496a-966e-6cfea38936e7-kube-api-access-5dxlv\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:43:08.589274 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:08.589279    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80e4bd69-feb8-496a-966e-6cfea38936e7-bundle\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:43:08.589511 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:08.589288    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd980ea9-8de2-47a5-ae4b-2e5b17764ff0-bundle\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:43:08.589511 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:08.589296    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5bb8\" (UniqueName: \"kubernetes.io/projected/fd980ea9-8de2-47a5-ae4b-2e5b17764ff0-kube-api-access-h5bb8\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:43:08.589511 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:08.589304    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd980ea9-8de2-47a5-ae4b-2e5b17764ff0-util\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:43:09.192435 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:09.192403    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp"
Jul 17 12:43:09.192435 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:09.192428    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp" event={"ID":"80e4bd69-feb8-496a-966e-6cfea38936e7","Type":"ContainerDied","Data":"186f8f68c26f96c84a2ab4f20d0c2a8eb7e10384338a446ca3d028d08c44c3e3"}
Jul 17 12:43:09.192695 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:09.192476    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="186f8f68c26f96c84a2ab4f20d0c2a8eb7e10384338a446ca3d028d08c44c3e3"
Jul 17 12:43:09.194470 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:09.194437    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg" event={"ID":"fd980ea9-8de2-47a5-ae4b-2e5b17764ff0","Type":"ContainerDied","Data":"5f76272432b2d2cf3a67f35976070405cfd3e3a3b906c79e9186562688a89b8e"}
Jul 17 12:43:09.194470 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:09.194460    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg"
Jul 17 12:43:09.194679 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:09.194490    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f76272432b2d2cf3a67f35976070405cfd3e3a3b906c79e9186562688a89b8e"
Jul 17 12:43:11.206544 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:11.206512    2565 generic.go:356] "Generic (PLEG): container finished" podID="488f355f-da9e-448e-b066-ed881e791765" containerID="d535fac1872806d055ec7d721f865add1855b29a074aaa3281a0928398f79c66" exitCode=0
Jul 17 12:43:11.206915 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:11.206551    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft" event={"ID":"488f355f-da9e-448e-b066-ed881e791765","Type":"ContainerDied","Data":"d535fac1872806d055ec7d721f865add1855b29a074aaa3281a0928398f79c66"}
Jul 17 12:43:12.213419 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:12.213367    2565 generic.go:356] "Generic (PLEG): container finished" podID="488f355f-da9e-448e-b066-ed881e791765" containerID="515f742b40bc9afddbf6d5d2e5109f101a7c8495178cd12c544db69b10441bb7" exitCode=0
Jul 17 12:43:12.213765 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:12.213451    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft" event={"ID":"488f355f-da9e-448e-b066-ed881e791765","Type":"ContainerDied","Data":"515f742b40bc9afddbf6d5d2e5109f101a7c8495178cd12c544db69b10441bb7"}
Jul 17 12:43:13.356727 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:13.356701    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft"
Jul 17 12:43:13.533158 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:13.533053    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/488f355f-da9e-448e-b066-ed881e791765-util\") pod \"488f355f-da9e-448e-b066-ed881e791765\" (UID: \"488f355f-da9e-448e-b066-ed881e791765\") "
Jul 17 12:43:13.533158 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:13.533113    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v776d\" (UniqueName: \"kubernetes.io/projected/488f355f-da9e-448e-b066-ed881e791765-kube-api-access-v776d\") pod \"488f355f-da9e-448e-b066-ed881e791765\" (UID: \"488f355f-da9e-448e-b066-ed881e791765\") "
Jul 17 12:43:13.533158 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:13.533140    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/488f355f-da9e-448e-b066-ed881e791765-bundle\") pod \"488f355f-da9e-448e-b066-ed881e791765\" (UID: \"488f355f-da9e-448e-b066-ed881e791765\") "
Jul 17 12:43:13.533712 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:13.533675    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488f355f-da9e-448e-b066-ed881e791765-bundle" (OuterVolumeSpecName: "bundle") pod "488f355f-da9e-448e-b066-ed881e791765" (UID: "488f355f-da9e-448e-b066-ed881e791765"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:43:13.535466 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:13.535445    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488f355f-da9e-448e-b066-ed881e791765-kube-api-access-v776d" (OuterVolumeSpecName: "kube-api-access-v776d") pod "488f355f-da9e-448e-b066-ed881e791765" (UID: "488f355f-da9e-448e-b066-ed881e791765"). InnerVolumeSpecName "kube-api-access-v776d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:43:13.537709 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:13.537690    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488f355f-da9e-448e-b066-ed881e791765-util" (OuterVolumeSpecName: "util") pod "488f355f-da9e-448e-b066-ed881e791765" (UID: "488f355f-da9e-448e-b066-ed881e791765"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:43:13.634573 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:13.634525    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/488f355f-da9e-448e-b066-ed881e791765-util\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:43:13.634573 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:13.634566    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v776d\" (UniqueName: \"kubernetes.io/projected/488f355f-da9e-448e-b066-ed881e791765-kube-api-access-v776d\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:43:13.634573 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:13.634576    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/488f355f-da9e-448e-b066-ed881e791765-bundle\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:43:14.224870 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:14.224841    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft"
Jul 17 12:43:14.224870 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:14.224856    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft" event={"ID":"488f355f-da9e-448e-b066-ed881e791765","Type":"ContainerDied","Data":"e6ad4a59e63e0a80fb72bf91cd8d0dce2a7f562e96e8d1e2f20089e30da49d0e"}
Jul 17 12:43:14.225101 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:14.224900    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6ad4a59e63e0a80fb72bf91cd8d0dce2a7f562e96e8d1e2f20089e30da49d0e"
Jul 17 12:43:19.353209 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:19.353161    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:20.748938 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:20.748907    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:20.981058 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:20.981021    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:21.089501 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:21.089464    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:23.430237 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430197    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-g7qk6"]
Jul 17 12:43:23.430709 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430690    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd980ea9-8de2-47a5-ae4b-2e5b17764ff0" containerName="extract"
Jul 17 12:43:23.430785 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430713    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd980ea9-8de2-47a5-ae4b-2e5b17764ff0" containerName="extract"
Jul 17 12:43:23.430785 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430729    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80e4bd69-feb8-496a-966e-6cfea38936e7" containerName="extract"
Jul 17 12:43:23.430785 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430738    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="80e4bd69-feb8-496a-966e-6cfea38936e7" containerName="extract"
Jul 17 12:43:23.430785 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430751    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80e4bd69-feb8-496a-966e-6cfea38936e7" containerName="pull"
Jul 17 12:43:23.430785 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430760    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="80e4bd69-feb8-496a-966e-6cfea38936e7" containerName="pull"
Jul 17 12:43:23.430785 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430771    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80e4bd69-feb8-496a-966e-6cfea38936e7" containerName="util"
Jul 17 12:43:23.430785 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430779    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="80e4bd69-feb8-496a-966e-6cfea38936e7" containerName="util"
Jul 17 12:43:23.431139 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430790    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b80e493-a458-4093-9173-922689bce863" containerName="pull"
Jul 17 12:43:23.431139 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430798    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b80e493-a458-4093-9173-922689bce863" containerName="pull"
Jul 17 12:43:23.431139 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430810    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b80e493-a458-4093-9173-922689bce863" containerName="util"
Jul 17 12:43:23.431139 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430819    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b80e493-a458-4093-9173-922689bce863" containerName="util"
Jul 17 12:43:23.431139 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430838    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd980ea9-8de2-47a5-ae4b-2e5b17764ff0" containerName="util"
Jul 17 12:43:23.431139 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430846    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd980ea9-8de2-47a5-ae4b-2e5b17764ff0" containerName="util"
Jul 17 12:43:23.431139 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430853    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="488f355f-da9e-448e-b066-ed881e791765" containerName="util"
Jul 17 12:43:23.431139 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430862    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="488f355f-da9e-448e-b066-ed881e791765" containerName="util"
Jul 17 12:43:23.431139 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430882    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd980ea9-8de2-47a5-ae4b-2e5b17764ff0" containerName="pull"
Jul 17 12:43:23.431139 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430890    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd980ea9-8de2-47a5-ae4b-2e5b17764ff0" containerName="pull"
Jul 17 12:43:23.431139 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430897    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="488f355f-da9e-448e-b066-ed881e791765" containerName="extract"
Jul 17 12:43:23.431139 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430905    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="488f355f-da9e-448e-b066-ed881e791765" containerName="extract"
Jul 17 12:43:23.431139 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430914    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="488f355f-da9e-448e-b066-ed881e791765" containerName="pull"
Jul 17 12:43:23.431139 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430922    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="488f355f-da9e-448e-b066-ed881e791765" containerName="pull"
Jul 17 12:43:23.431139 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430935    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b80e493-a458-4093-9173-922689bce863" containerName="extract"
Jul 17 12:43:23.431139 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.430943    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b80e493-a458-4093-9173-922689bce863" containerName="extract"
Jul 17 12:43:23.431139 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.431044    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="488f355f-da9e-448e-b066-ed881e791765" containerName="extract"
Jul 17 12:43:23.431139 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.431059    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="80e4bd69-feb8-496a-966e-6cfea38936e7" containerName="extract"
Jul 17 12:43:23.431139 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.431070    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd980ea9-8de2-47a5-ae4b-2e5b17764ff0" containerName="extract"
Jul 17 12:43:23.431139 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.431080    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b80e493-a458-4093-9173-922689bce863" containerName="extract"
Jul 17 12:43:23.437285 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.437267    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-g7qk6"
Jul 17 12:43:23.440556 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.440530    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-kxv2r\""
Jul 17 12:43:23.450674 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.450641    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-g7qk6"]
Jul 17 12:43:23.518524 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.518488    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qmmg\" (UniqueName: \"kubernetes.io/projected/c67bff4e-9316-4494-ab0b-3115a5b59e82-kube-api-access-9qmmg\") pod \"authorino-operator-75cc8d5cb7-g7qk6\" (UID: \"c67bff4e-9316-4494-ab0b-3115a5b59e82\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-g7qk6"
Jul 17 12:43:23.619840 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.619805    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qmmg\" (UniqueName: \"kubernetes.io/projected/c67bff4e-9316-4494-ab0b-3115a5b59e82-kube-api-access-9qmmg\") pod \"authorino-operator-75cc8d5cb7-g7qk6\" (UID: \"c67bff4e-9316-4494-ab0b-3115a5b59e82\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-g7qk6"
Jul 17 12:43:23.628774 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.628751    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qmmg\" (UniqueName: \"kubernetes.io/projected/c67bff4e-9316-4494-ab0b-3115a5b59e82-kube-api-access-9qmmg\") pod \"authorino-operator-75cc8d5cb7-g7qk6\" (UID: \"c67bff4e-9316-4494-ab0b-3115a5b59e82\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-g7qk6"
Jul 17 12:43:23.760833 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.760764    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-g7qk6"
Jul 17 12:43:23.893782 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:23.893758    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-g7qk6"]
Jul 17 12:43:23.895824 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:43:23.895795    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67bff4e_9316_4494_ab0b_3115a5b59e82.slice/crio-1ba74f827b5b06dffcce9368a85d718f132cca5385e0dd1531270cc3066a43ca WatchSource:0}: Error finding container 1ba74f827b5b06dffcce9368a85d718f132cca5385e0dd1531270cc3066a43ca: Status 404 returned error can't find the container with id 1ba74f827b5b06dffcce9368a85d718f132cca5385e0dd1531270cc3066a43ca
Jul 17 12:43:24.275412 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:24.275376    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-g7qk6" event={"ID":"c67bff4e-9316-4494-ab0b-3115a5b59e82","Type":"ContainerStarted","Data":"1ba74f827b5b06dffcce9368a85d718f132cca5385e0dd1531270cc3066a43ca"}
Jul 17 12:43:24.566589 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:24.566555    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:26.288265 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:26.288237    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-g7qk6" event={"ID":"c67bff4e-9316-4494-ab0b-3115a5b59e82","Type":"ContainerStarted","Data":"ed8e1a1b470b30773fb51875e530c5a43cb464fc21081d29d2ea28e89c0fd3b7"}
Jul 17 12:43:26.288611 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:26.288370    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-g7qk6"
Jul 17 12:43:26.308196 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:26.308144    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-g7qk6" podStartSLOduration=1.0486891 podStartE2EDuration="3.308126834s" totalImagesPullingTime="2.259437734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:43:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:43:26.304901985 +0000 UTC m=+1250.436286660" watchObservedRunningTime="2026-07-17 12:43:26.308126834 +0000 UTC m=+1250.439511497"
Jul 17 12:43:26.479095 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:26.479020    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh"]
Jul 17 12:43:26.482415 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:26.482400    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh"
Jul 17 12:43:26.485763 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:26.485744    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-6g99p\""
Jul 17 12:43:26.493909 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:26.493887    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh"]
Jul 17 12:43:26.548020 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:26.547976    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sbvx\" (UniqueName: \"kubernetes.io/projected/0be23306-8e45-437a-bd80-82aa5ad3aa70-kube-api-access-2sbvx\") pod \"limitador-operator-controller-manager-859d7b78bd-hxlqh\" (UID: \"0be23306-8e45-437a-bd80-82aa5ad3aa70\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh"
Jul 17 12:43:26.648895 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:26.648864    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sbvx\" (UniqueName: \"kubernetes.io/projected/0be23306-8e45-437a-bd80-82aa5ad3aa70-kube-api-access-2sbvx\") pod \"limitador-operator-controller-manager-859d7b78bd-hxlqh\" (UID: \"0be23306-8e45-437a-bd80-82aa5ad3aa70\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh"
Jul 17 12:43:26.658369 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:26.658343    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sbvx\" (UniqueName: \"kubernetes.io/projected/0be23306-8e45-437a-bd80-82aa5ad3aa70-kube-api-access-2sbvx\") pod \"limitador-operator-controller-manager-859d7b78bd-hxlqh\" (UID: \"0be23306-8e45-437a-bd80-82aa5ad3aa70\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh"
Jul 17 12:43:26.793973 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:26.793880    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh"
Jul 17 12:43:26.940368 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:43:26.940342    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be23306_8e45_437a_bd80_82aa5ad3aa70.slice/crio-789ea574822a1f9a3adb3e575048b68ff4140734e2b40d649dc7725d2abf258a WatchSource:0}: Error finding container 789ea574822a1f9a3adb3e575048b68ff4140734e2b40d649dc7725d2abf258a: Status 404 returned error can't find the container with id 789ea574822a1f9a3adb3e575048b68ff4140734e2b40d649dc7725d2abf258a
Jul 17 12:43:26.941108 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:26.941072    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh"]
Jul 17 12:43:27.039787 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:27.039755    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:27.298156 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:27.298124    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh" event={"ID":"0be23306-8e45-437a-bd80-82aa5ad3aa70","Type":"ContainerStarted","Data":"789ea574822a1f9a3adb3e575048b68ff4140734e2b40d649dc7725d2abf258a"}
Jul 17 12:43:29.310719 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:29.310687    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh" event={"ID":"0be23306-8e45-437a-bd80-82aa5ad3aa70","Type":"ContainerStarted","Data":"876d6f7a437231201f89cb09fcaf6cbd0f5758a6b58af28e9f339d9c302a9d8a"}
Jul 17 12:43:29.311088 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:29.310743    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh"
Jul 17 12:43:29.329878 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:29.329830    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh" podStartSLOduration=1.737427801 podStartE2EDuration="3.329816683s" totalImagesPullingTime="1.592388882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:43:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:43:29.328826188 +0000 UTC m=+1253.460210852" watchObservedRunningTime="2026-07-17 12:43:29.329816683 +0000 UTC m=+1253.461201341"
Jul 17 12:43:29.804299 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:29.804267    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:29.911821 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:29.911781    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:30.188471 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:30.188393    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"]
Jul 17 12:43:30.191929 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:30.191914    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"
Jul 17 12:43:30.195203 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:30.195185    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qv7t7\""
Jul 17 12:43:30.221792 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:30.221768    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"]
Jul 17 12:43:30.281491 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:30.281462    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b1a2a61-8e79-4cf1-94c2-70c7f52843fb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nd5tp\" (UID: \"5b1a2a61-8e79-4cf1-94c2-70c7f52843fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"
Jul 17 12:43:30.281647 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:30.281535    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmhbv\" (UniqueName: \"kubernetes.io/projected/5b1a2a61-8e79-4cf1-94c2-70c7f52843fb-kube-api-access-tmhbv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nd5tp\" (UID: \"5b1a2a61-8e79-4cf1-94c2-70c7f52843fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"
Jul 17 12:43:30.382345 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:30.382308    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b1a2a61-8e79-4cf1-94c2-70c7f52843fb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nd5tp\" (UID: \"5b1a2a61-8e79-4cf1-94c2-70c7f52843fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"
Jul 17 12:43:30.382693 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:30.382418    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmhbv\" (UniqueName: \"kubernetes.io/projected/5b1a2a61-8e79-4cf1-94c2-70c7f52843fb-kube-api-access-tmhbv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nd5tp\" (UID: \"5b1a2a61-8e79-4cf1-94c2-70c7f52843fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"
Jul 17 12:43:30.382757 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:30.382733    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b1a2a61-8e79-4cf1-94c2-70c7f52843fb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nd5tp\" (UID: \"5b1a2a61-8e79-4cf1-94c2-70c7f52843fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"
Jul 17 12:43:30.393803 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:30.393772    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmhbv\" (UniqueName: \"kubernetes.io/projected/5b1a2a61-8e79-4cf1-94c2-70c7f52843fb-kube-api-access-tmhbv\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nd5tp\" (UID: \"5b1a2a61-8e79-4cf1-94c2-70c7f52843fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"
Jul 17 12:43:30.501827 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:30.501748    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"
Jul 17 12:43:30.634164 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:30.634136    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"]
Jul 17 12:43:30.635402 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:43:30.635376    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1a2a61_8e79_4cf1_94c2_70c7f52843fb.slice/crio-1854ecdd868a6b5eab71b78483fc23a97b9e8cd133f68f8afdff2734aaadeeca WatchSource:0}: Error finding container 1854ecdd868a6b5eab71b78483fc23a97b9e8cd133f68f8afdff2734aaadeeca: Status 404 returned error can't find the container with id 1854ecdd868a6b5eab71b78483fc23a97b9e8cd133f68f8afdff2734aaadeeca
Jul 17 12:43:30.785638 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:30.785544    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:31.324549 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:31.324511    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp" event={"ID":"5b1a2a61-8e79-4cf1-94c2-70c7f52843fb","Type":"ContainerStarted","Data":"1854ecdd868a6b5eab71b78483fc23a97b9e8cd133f68f8afdff2734aaadeeca"}
Jul 17 12:43:32.358283 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:32.358243    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5pwp"]
Jul 17 12:43:32.361569 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:32.361547    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5pwp"
Jul 17 12:43:32.364496 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:32.364475    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-6t5rv\""
Jul 17 12:43:32.364646 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:32.364476    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 17 12:43:32.378303 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:32.378279    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5pwp"]
Jul 17 12:43:32.400205 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:32.400178    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f7sm\" (UniqueName: \"kubernetes.io/projected/0863561d-bd8f-43e9-ad0c-9e862c8ea29d-kube-api-access-6f7sm\") pod \"dns-operator-controller-manager-864dd78c4c-p5pwp\" (UID: \"0863561d-bd8f-43e9-ad0c-9e862c8ea29d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5pwp"
Jul 17 12:43:32.501668 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:32.501638    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f7sm\" (UniqueName: \"kubernetes.io/projected/0863561d-bd8f-43e9-ad0c-9e862c8ea29d-kube-api-access-6f7sm\") pod \"dns-operator-controller-manager-864dd78c4c-p5pwp\" (UID: \"0863561d-bd8f-43e9-ad0c-9e862c8ea29d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5pwp"
Jul 17 12:43:32.511335 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:32.511305    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f7sm\" (UniqueName: \"kubernetes.io/projected/0863561d-bd8f-43e9-ad0c-9e862c8ea29d-kube-api-access-6f7sm\") pod \"dns-operator-controller-manager-864dd78c4c-p5pwp\" (UID: \"0863561d-bd8f-43e9-ad0c-9e862c8ea29d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5pwp"
Jul 17 12:43:32.676022 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:32.675911    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5pwp"
Jul 17 12:43:32.827111 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:32.827075    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5pwp"]
Jul 17 12:43:32.827943 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:43:32.827915    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0863561d_bd8f_43e9_ad0c_9e862c8ea29d.slice/crio-d3df73fab25ab35341dd34e2e2aec728a1b85739032839dc38d14bb8d45e1dd2 WatchSource:0}: Error finding container d3df73fab25ab35341dd34e2e2aec728a1b85739032839dc38d14bb8d45e1dd2: Status 404 returned error can't find the container with id d3df73fab25ab35341dd34e2e2aec728a1b85739032839dc38d14bb8d45e1dd2
Jul 17 12:43:33.341700 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:33.341657    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5pwp" event={"ID":"0863561d-bd8f-43e9-ad0c-9e862c8ea29d","Type":"ContainerStarted","Data":"d3df73fab25ab35341dd34e2e2aec728a1b85739032839dc38d14bb8d45e1dd2"}
Jul 17 12:43:33.342293 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:33.342250    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:33.766462 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:33.766391    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:33.920364 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:33.920331    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:34.158763 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:34.158701    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:34.228044 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:34.228012    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:36.363603 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:36.363566    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp" event={"ID":"5b1a2a61-8e79-4cf1-94c2-70c7f52843fb","Type":"ContainerStarted","Data":"3449ff5ce6cac24f7ee1259befa8ad9f5c1d1ab8e855b7f134be28c59720ef76"}
Jul 17 12:43:36.364014 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:36.363759    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"
Jul 17 12:43:36.365241 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:36.365219    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5pwp" event={"ID":"0863561d-bd8f-43e9-ad0c-9e862c8ea29d","Type":"ContainerStarted","Data":"bd2641d7bb58a23eded7fe0db5e0082c86f6f1bee1138771909a2501453b981f"}
Jul 17 12:43:36.365333 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:36.365268    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5pwp"
Jul 17 12:43:36.384034 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:36.383963    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp" podStartSLOduration=1.056939283 podStartE2EDuration="6.383949621s" totalImagesPullingTime="5.327010338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:43:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:43:36.382228967 +0000 UTC m=+1260.513613643" watchObservedRunningTime="2026-07-17 12:43:36.383949621 +0000 UTC m=+1260.515334283"
Jul 17 12:43:36.401193 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:36.401145    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5pwp" podStartSLOduration=1.270506912 podStartE2EDuration="4.401128995s" totalImagesPullingTime="3.130622083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:43:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:43:36.399470481 +0000 UTC m=+1260.530855143" watchObservedRunningTime="2026-07-17 12:43:36.401128995 +0000 UTC m=+1260.532513657"
Jul 17 12:43:37.300681 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:37.300646    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-g7qk6"
Jul 17 12:43:37.361571 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:37.361537    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:37.467850 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:37.467812    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:38.833848 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:38.833816    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-4pvhs"]
Jul 17 12:43:38.838545 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:38.838527    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4pvhs"
Jul 17 12:43:38.841442 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:38.841418    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 17 12:43:38.841554 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:38.841440    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 17 12:43:38.841630 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:38.841565    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mb4gp\""
Jul 17 12:43:38.847932 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:38.847914    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-4pvhs"]
Jul 17 12:43:38.851350 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:38.851324    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:38.959364 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:38.959342    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f46a17f9-aba8-4257-b052-f5cf6f87d5cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-4pvhs\" (UID: \"f46a17f9-aba8-4257-b052-f5cf6f87d5cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4pvhs"
Jul 17 12:43:38.959468 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:38.959402    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmttp\" (UniqueName: \"kubernetes.io/projected/f46a17f9-aba8-4257-b052-f5cf6f87d5cc-kube-api-access-tmttp\") pod \"kuadrant-console-plugin-66b5774985-4pvhs\" (UID: \"f46a17f9-aba8-4257-b052-f5cf6f87d5cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4pvhs"
Jul 17 12:43:38.959468 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:38.959429    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f46a17f9-aba8-4257-b052-f5cf6f87d5cc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-4pvhs\" (UID: \"f46a17f9-aba8-4257-b052-f5cf6f87d5cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4pvhs"
Jul 17 12:43:39.059898 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:39.059869    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmttp\" (UniqueName: \"kubernetes.io/projected/f46a17f9-aba8-4257-b052-f5cf6f87d5cc-kube-api-access-tmttp\") pod \"kuadrant-console-plugin-66b5774985-4pvhs\" (UID: \"f46a17f9-aba8-4257-b052-f5cf6f87d5cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4pvhs"
Jul 17 12:43:39.060027 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:39.059902    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f46a17f9-aba8-4257-b052-f5cf6f87d5cc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-4pvhs\" (UID: \"f46a17f9-aba8-4257-b052-f5cf6f87d5cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4pvhs"
Jul 17 12:43:39.060027 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:39.059947    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f46a17f9-aba8-4257-b052-f5cf6f87d5cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-4pvhs\" (UID: \"f46a17f9-aba8-4257-b052-f5cf6f87d5cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4pvhs"
Jul 17 12:43:39.060195 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:43:39.060035    2565 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 17 12:43:39.060195 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:43:39.060086    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f46a17f9-aba8-4257-b052-f5cf6f87d5cc-plugin-serving-cert podName:f46a17f9-aba8-4257-b052-f5cf6f87d5cc nodeName:}" failed. No retries permitted until 2026-07-17 12:43:39.560072155 +0000 UTC m=+1263.691456794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/f46a17f9-aba8-4257-b052-f5cf6f87d5cc-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-4pvhs" (UID: "f46a17f9-aba8-4257-b052-f5cf6f87d5cc") : secret "plugin-serving-cert" not found
Jul 17 12:43:39.060505 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:39.060487    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f46a17f9-aba8-4257-b052-f5cf6f87d5cc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-4pvhs\" (UID: \"f46a17f9-aba8-4257-b052-f5cf6f87d5cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4pvhs"
Jul 17 12:43:39.076397 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:39.076368    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmttp\" (UniqueName: \"kubernetes.io/projected/f46a17f9-aba8-4257-b052-f5cf6f87d5cc-kube-api-access-tmttp\") pod \"kuadrant-console-plugin-66b5774985-4pvhs\" (UID: \"f46a17f9-aba8-4257-b052-f5cf6f87d5cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4pvhs"
Jul 17 12:43:39.209222 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:39.209159    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:39.563001 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:39.562957    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f46a17f9-aba8-4257-b052-f5cf6f87d5cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-4pvhs\" (UID: \"f46a17f9-aba8-4257-b052-f5cf6f87d5cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4pvhs"
Jul 17 12:43:39.565475 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:39.565449    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/f46a17f9-aba8-4257-b052-f5cf6f87d5cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-4pvhs\" (UID: \"f46a17f9-aba8-4257-b052-f5cf6f87d5cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4pvhs"
Jul 17 12:43:39.748926 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:39.748895    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4pvhs"
Jul 17 12:43:39.885097 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:39.885075    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-4pvhs"]
Jul 17 12:43:39.886581 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:43:39.886560    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf46a17f9_aba8_4257_b052_f5cf6f87d5cc.slice/crio-1d3d6bb9d129ea41e90bac22d8fe3ed3794f76a32da2f1c776f84b0c263d3476 WatchSource:0}: Error finding container 1d3d6bb9d129ea41e90bac22d8fe3ed3794f76a32da2f1c776f84b0c263d3476: Status 404 returned error can't find the container with id 1d3d6bb9d129ea41e90bac22d8fe3ed3794f76a32da2f1c776f84b0c263d3476
Jul 17 12:43:40.319308 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:40.319278    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh"
Jul 17 12:43:40.387732 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:40.387700    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4pvhs" event={"ID":"f46a17f9-aba8-4257-b052-f5cf6f87d5cc","Type":"ContainerStarted","Data":"1d3d6bb9d129ea41e90bac22d8fe3ed3794f76a32da2f1c776f84b0c263d3476"}
Jul 17 12:43:40.858808 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:40.858773    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:40.983754 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:40.983714    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:46.571785 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:46.571751    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:46.672802 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:46.672774    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:47.372229 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:47.372198    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"
Jul 17 12:43:47.372409 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:47.372296    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p5pwp"
Jul 17 12:43:49.329618 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.329579    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"]
Jul 17 12:43:49.330075 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.329843    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp" podUID="5b1a2a61-8e79-4cf1-94c2-70c7f52843fb" containerName="manager" containerID="cri-o://3449ff5ce6cac24f7ee1259befa8ad9f5c1d1ab8e855b7f134be28c59720ef76" gracePeriod=2
Jul 17 12:43:49.339260 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.339179    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"]
Jul 17 12:43:49.351501 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:43:49.351460    2565 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"5b1a2a61-8e79-4cf1-94c2-70c7f52843fb\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-nd5tp\": pods \"kuadrant-operator-controller-manager-8684c9ff49-nd5tp\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"
Jul 17 12:43:49.355083 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.355056    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh"]
Jul 17 12:43:49.355461 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.355430    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh" podUID="0be23306-8e45-437a-bd80-82aa5ad3aa70" containerName="manager" containerID="cri-o://876d6f7a437231201f89cb09fcaf6cbd0f5758a6b58af28e9f339d9c302a9d8a" gracePeriod=2
Jul 17 12:43:49.358514 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:43:49.358488    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nd5tp\" is forbidden: User \"system:node:ip-10-0-134-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-119.ec2.internal' and this object" podUID="5b1a2a61-8e79-4cf1-94c2-70c7f52843fb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"
Jul 17 12:43:49.364439 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.364152    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7"]
Jul 17 12:43:49.366090 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.364728    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b1a2a61-8e79-4cf1-94c2-70c7f52843fb" containerName="manager"
Jul 17 12:43:49.366090 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.364752    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b1a2a61-8e79-4cf1-94c2-70c7f52843fb" containerName="manager"
Jul 17 12:43:49.366090 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.364873    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b1a2a61-8e79-4cf1-94c2-70c7f52843fb" containerName="manager"
Jul 17 12:43:49.368410 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.368388    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh"]
Jul 17 12:43:49.368616 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.368496    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7"
Jul 17 12:43:49.374454 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:43:49.374421    2565 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0be23306-8e45-437a-bd80-82aa5ad3aa70\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-hxlqh\": pods \"limitador-operator-controller-manager-859d7b78bd-hxlqh\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh"
Jul 17 12:43:49.378201 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:43:49.378171    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-hxlqh\" is forbidden: User \"system:node:ip-10-0-134-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-119.ec2.internal' and this object" podUID="0be23306-8e45-437a-bd80-82aa5ad3aa70" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh"
Jul 17 12:43:49.382332 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:43:49.382305    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nd5tp\" is forbidden: User \"system:node:ip-10-0-134-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-119.ec2.internal' and this object" podUID="5b1a2a61-8e79-4cf1-94c2-70c7f52843fb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"
Jul 17 12:43:49.384279 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.384185    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7q6d"]
Jul 17 12:43:49.385067 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.384836    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0be23306-8e45-437a-bd80-82aa5ad3aa70" containerName="manager"
Jul 17 12:43:49.385067 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.384858    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0be23306-8e45-437a-bd80-82aa5ad3aa70" containerName="manager"
Jul 17 12:43:49.385067 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.384948    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0be23306-8e45-437a-bd80-82aa5ad3aa70" containerName="manager"
Jul 17 12:43:49.388368 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.388317    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7"]
Jul 17 12:43:49.388464 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.388436    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7q6d"
Jul 17 12:43:49.398553 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.398393    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7q6d"]
Jul 17 12:43:49.410708 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:43:49.410672    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-hxlqh\" is forbidden: User \"system:node:ip-10-0-134-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-119.ec2.internal' and this object" podUID="0be23306-8e45-437a-bd80-82aa5ad3aa70" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh"
Jul 17 12:43:49.443139 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:43:49.443100    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nd5tp\" is forbidden: User \"system:node:ip-10-0-134-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-119.ec2.internal' and this object" podUID="5b1a2a61-8e79-4cf1-94c2-70c7f52843fb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"
Jul 17 12:43:49.456217 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.456192    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/426e349a-ef6d-4ad9-afca-8aa6104b3f00-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tqsr7\" (UID: \"426e349a-ef6d-4ad9-afca-8aa6104b3f00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7"
Jul 17 12:43:49.456378 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.456249    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rnt8\" (UniqueName: \"kubernetes.io/projected/4182c07e-dbcf-4223-8253-da988273a232-kube-api-access-5rnt8\") pod \"limitador-operator-controller-manager-859d7b78bd-h7q6d\" (UID: \"4182c07e-dbcf-4223-8253-da988273a232\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7q6d"
Jul 17 12:43:49.456378 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.456346    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj42h\" (UniqueName: \"kubernetes.io/projected/426e349a-ef6d-4ad9-afca-8aa6104b3f00-kube-api-access-nj42h\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tqsr7\" (UID: \"426e349a-ef6d-4ad9-afca-8aa6104b3f00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7"
Jul 17 12:43:49.557779 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.557749    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/426e349a-ef6d-4ad9-afca-8aa6104b3f00-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tqsr7\" (UID: \"426e349a-ef6d-4ad9-afca-8aa6104b3f00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7"
Jul 17 12:43:49.557948 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.557798    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rnt8\" (UniqueName: \"kubernetes.io/projected/4182c07e-dbcf-4223-8253-da988273a232-kube-api-access-5rnt8\") pod \"limitador-operator-controller-manager-859d7b78bd-h7q6d\" (UID: \"4182c07e-dbcf-4223-8253-da988273a232\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7q6d"
Jul 17 12:43:49.557948 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.557855    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj42h\" (UniqueName: \"kubernetes.io/projected/426e349a-ef6d-4ad9-afca-8aa6104b3f00-kube-api-access-nj42h\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tqsr7\" (UID: \"426e349a-ef6d-4ad9-afca-8aa6104b3f00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7"
Jul 17 12:43:49.558211 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.558186    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/426e349a-ef6d-4ad9-afca-8aa6104b3f00-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tqsr7\" (UID: \"426e349a-ef6d-4ad9-afca-8aa6104b3f00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7"
Jul 17 12:43:49.573756 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.573732    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj42h\" (UniqueName: \"kubernetes.io/projected/426e349a-ef6d-4ad9-afca-8aa6104b3f00-kube-api-access-nj42h\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tqsr7\" (UID: \"426e349a-ef6d-4ad9-afca-8aa6104b3f00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7"
Jul 17 12:43:49.573929 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.573905    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rnt8\" (UniqueName: \"kubernetes.io/projected/4182c07e-dbcf-4223-8253-da988273a232-kube-api-access-5rnt8\") pod \"limitador-operator-controller-manager-859d7b78bd-h7q6d\" (UID: \"4182c07e-dbcf-4223-8253-da988273a232\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7q6d"
Jul 17 12:43:49.579679 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.579624    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl"]
Jul 17 12:43:49.585513 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.585493    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl"
Jul 17 12:43:49.602763 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.602737    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl"]
Jul 17 12:43:49.624016 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:43:49.623958    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nd5tp\" is forbidden: User \"system:node:ip-10-0-134-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-119.ec2.internal' and this object" podUID="5b1a2a61-8e79-4cf1-94c2-70c7f52843fb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"
Jul 17 12:43:49.626546 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:43:49.626515    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-hxlqh\" is forbidden: User \"system:node:ip-10-0-134-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-119.ec2.internal' and this object" podUID="0be23306-8e45-437a-bd80-82aa5ad3aa70" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh"
Jul 17 12:43:49.658933 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.658909    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50f68e7a-3d6a-4da2-9264-06fec2b550e6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hlkvl\" (UID: \"50f68e7a-3d6a-4da2-9264-06fec2b550e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl"
Jul 17 12:43:49.659101 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.658956    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47fd6\" (UniqueName: \"kubernetes.io/projected/50f68e7a-3d6a-4da2-9264-06fec2b550e6-kube-api-access-47fd6\") pod \"kuadrant-operator-controller-manager-86878c769f-hlkvl\" (UID: \"50f68e7a-3d6a-4da2-9264-06fec2b550e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl"
Jul 17 12:43:49.743541 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.743514    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7"
Jul 17 12:43:49.751801 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.751780    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7q6d"
Jul 17 12:43:49.760091 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.760067    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50f68e7a-3d6a-4da2-9264-06fec2b550e6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hlkvl\" (UID: \"50f68e7a-3d6a-4da2-9264-06fec2b550e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl"
Jul 17 12:43:49.760204 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.760125    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47fd6\" (UniqueName: \"kubernetes.io/projected/50f68e7a-3d6a-4da2-9264-06fec2b550e6-kube-api-access-47fd6\") pod \"kuadrant-operator-controller-manager-86878c769f-hlkvl\" (UID: \"50f68e7a-3d6a-4da2-9264-06fec2b550e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl"
Jul 17 12:43:49.760456 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.760435    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50f68e7a-3d6a-4da2-9264-06fec2b550e6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hlkvl\" (UID: \"50f68e7a-3d6a-4da2-9264-06fec2b550e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl"
Jul 17 12:43:49.770292 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.770273    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47fd6\" (UniqueName: \"kubernetes.io/projected/50f68e7a-3d6a-4da2-9264-06fec2b550e6-kube-api-access-47fd6\") pod \"kuadrant-operator-controller-manager-86878c769f-hlkvl\" (UID: \"50f68e7a-3d6a-4da2-9264-06fec2b550e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl"
Jul 17 12:43:49.907367 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:49.907284    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl"
Jul 17 12:43:51.318374 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:51.318330    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh" podUID="0be23306-8e45-437a-bd80-82aa5ad3aa70" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.40:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 17 12:43:52.315309 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:52.315256    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:52.417521 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:52.417487    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:52.941369 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:52.941338    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:53.168156 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:53.168120    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:53.562295 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:53.562255    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:53.662357 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:53.662322    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:56.082159 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:56.081972    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:56.150866 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:56.150825    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:56.719228 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:56.719190    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:43:56.857792 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:43:56.857695    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:44:00.120965 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:00.120927    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:44:00.226915 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:00.226883    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:44:01.001036 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.000711    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"
Jul 17 12:44:01.003832 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:44:01.003783    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nd5tp\" is forbidden: User \"system:node:ip-10-0-134-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-119.ec2.internal' and this object" podUID="5b1a2a61-8e79-4cf1-94c2-70c7f52843fb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"
Jul 17 12:44:01.030586 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.030431    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh"
Jul 17 12:44:01.031255 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.031219    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl"]
Jul 17 12:44:01.032425 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:44:01.032390    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f68e7a_3d6a_4da2_9264_06fec2b550e6.slice/crio-ddcbf45b03a6d2281caa28c07de8c1ddbcea13d3148d999db68306d95eecec61 WatchSource:0}: Error finding container ddcbf45b03a6d2281caa28c07de8c1ddbcea13d3148d999db68306d95eecec61: Status 404 returned error can't find the container with id ddcbf45b03a6d2281caa28c07de8c1ddbcea13d3148d999db68306d95eecec61
Jul 17 12:44:01.032938 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:44:01.032910    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nd5tp\" is forbidden: User \"system:node:ip-10-0-134-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-119.ec2.internal' and this object" podUID="5b1a2a61-8e79-4cf1-94c2-70c7f52843fb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"
Jul 17 12:44:01.035291 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:44:01.035258    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-hxlqh\" is forbidden: User \"system:node:ip-10-0-134-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-119.ec2.internal' and this object" podUID="0be23306-8e45-437a-bd80-82aa5ad3aa70" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh"
Jul 17 12:44:01.063168 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.063147    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7"]
Jul 17 12:44:01.066170 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:44:01.066146    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426e349a_ef6d_4ad9_afca_8aa6104b3f00.slice/crio-874f73e72e834ed085b37e27ee9877f23ec4a7eceb0adc31280e09dd31fca71f WatchSource:0}: Error finding container 874f73e72e834ed085b37e27ee9877f23ec4a7eceb0adc31280e09dd31fca71f: Status 404 returned error can't find the container with id 874f73e72e834ed085b37e27ee9877f23ec4a7eceb0adc31280e09dd31fca71f
Jul 17 12:44:01.067700 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:44:01.067671    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4182c07e_dbcf_4223_8253_da988273a232.slice/crio-e7e42523d8b6553af24fa4c71061f159bec5f458f021cdb5614dccce67cf1a19 WatchSource:0}: Error finding container e7e42523d8b6553af24fa4c71061f159bec5f458f021cdb5614dccce67cf1a19: Status 404 returned error can't find the container with id e7e42523d8b6553af24fa4c71061f159bec5f458f021cdb5614dccce67cf1a19
Jul 17 12:44:01.070742 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.070720    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7q6d"]
Jul 17 12:44:01.168674 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.168649    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b1a2a61-8e79-4cf1-94c2-70c7f52843fb-extensions-socket-volume\") pod \"5b1a2a61-8e79-4cf1-94c2-70c7f52843fb\" (UID: \"5b1a2a61-8e79-4cf1-94c2-70c7f52843fb\") "
Jul 17 12:44:01.169073 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.168698    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sbvx\" (UniqueName: \"kubernetes.io/projected/0be23306-8e45-437a-bd80-82aa5ad3aa70-kube-api-access-2sbvx\") pod \"0be23306-8e45-437a-bd80-82aa5ad3aa70\" (UID: \"0be23306-8e45-437a-bd80-82aa5ad3aa70\") "
Jul 17 12:44:01.169073 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.168744    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmhbv\" (UniqueName: \"kubernetes.io/projected/5b1a2a61-8e79-4cf1-94c2-70c7f52843fb-kube-api-access-tmhbv\") pod \"5b1a2a61-8e79-4cf1-94c2-70c7f52843fb\" (UID: \"5b1a2a61-8e79-4cf1-94c2-70c7f52843fb\") "
Jul 17 12:44:01.169165 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.169096    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b1a2a61-8e79-4cf1-94c2-70c7f52843fb-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5b1a2a61-8e79-4cf1-94c2-70c7f52843fb" (UID: "5b1a2a61-8e79-4cf1-94c2-70c7f52843fb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:44:01.170790 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.170768    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be23306-8e45-437a-bd80-82aa5ad3aa70-kube-api-access-2sbvx" (OuterVolumeSpecName: "kube-api-access-2sbvx") pod "0be23306-8e45-437a-bd80-82aa5ad3aa70" (UID: "0be23306-8e45-437a-bd80-82aa5ad3aa70"). InnerVolumeSpecName "kube-api-access-2sbvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:44:01.170855 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.170819    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1a2a61-8e79-4cf1-94c2-70c7f52843fb-kube-api-access-tmhbv" (OuterVolumeSpecName: "kube-api-access-tmhbv") pod "5b1a2a61-8e79-4cf1-94c2-70c7f52843fb" (UID: "5b1a2a61-8e79-4cf1-94c2-70c7f52843fb"). InnerVolumeSpecName "kube-api-access-tmhbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:44:01.269309 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.269282    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b1a2a61-8e79-4cf1-94c2-70c7f52843fb-extensions-socket-volume\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:44:01.269309 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.269305    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sbvx\" (UniqueName: \"kubernetes.io/projected/0be23306-8e45-437a-bd80-82aa5ad3aa70-kube-api-access-2sbvx\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:44:01.269309 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.269314    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmhbv\" (UniqueName: \"kubernetes.io/projected/5b1a2a61-8e79-4cf1-94c2-70c7f52843fb-kube-api-access-tmhbv\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:44:01.526155 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.526067    2565 generic.go:356] "Generic (PLEG): container finished" podID="0be23306-8e45-437a-bd80-82aa5ad3aa70" containerID="876d6f7a437231201f89cb09fcaf6cbd0f5758a6b58af28e9f339d9c302a9d8a" exitCode=0
Jul 17 12:44:01.526155 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.526125    2565 scope.go:117] "RemoveContainer" containerID="876d6f7a437231201f89cb09fcaf6cbd0f5758a6b58af28e9f339d9c302a9d8a"
Jul 17 12:44:01.526379 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.526128    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh"
Jul 17 12:44:01.528060 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.528031    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4pvhs" event={"ID":"f46a17f9-aba8-4257-b052-f5cf6f87d5cc","Type":"ContainerStarted","Data":"f7bd253020c6046d58b893da3dddd1b3d54777d5408b5cde88ae9a1e801ba20f"}
Jul 17 12:44:01.529116 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:44:01.529022    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nd5tp\" is forbidden: User \"system:node:ip-10-0-134-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-119.ec2.internal' and this object" podUID="5b1a2a61-8e79-4cf1-94c2-70c7f52843fb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"
Jul 17 12:44:01.529846 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.529819    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7" event={"ID":"426e349a-ef6d-4ad9-afca-8aa6104b3f00","Type":"ContainerStarted","Data":"764c5e89938097b51b3ddce7e304e306fa9f1d0ac77fa108b50e5d87492670b7"}
Jul 17 12:44:01.529966 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.529870    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7"
Jul 17 12:44:01.529966 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.529893    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7" event={"ID":"426e349a-ef6d-4ad9-afca-8aa6104b3f00","Type":"ContainerStarted","Data":"874f73e72e834ed085b37e27ee9877f23ec4a7eceb0adc31280e09dd31fca71f"}
Jul 17 12:44:01.531157 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:44:01.531128    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-hxlqh\" is forbidden: User \"system:node:ip-10-0-134-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-119.ec2.internal' and this object" podUID="0be23306-8e45-437a-bd80-82aa5ad3aa70" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh"
Jul 17 12:44:01.531299 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.531243    2565 generic.go:356] "Generic (PLEG): container finished" podID="5b1a2a61-8e79-4cf1-94c2-70c7f52843fb" containerID="3449ff5ce6cac24f7ee1259befa8ad9f5c1d1ab8e855b7f134be28c59720ef76" exitCode=0
Jul 17 12:44:01.531299 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.531289    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"
Jul 17 12:44:01.533033 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.533013    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl" event={"ID":"50f68e7a-3d6a-4da2-9264-06fec2b550e6","Type":"ContainerStarted","Data":"153344f5522650cef149454e41d0fd12043ca807fc091d52f01b19acc48fcd3f"}
Jul 17 12:44:01.533132 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.533056    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl" event={"ID":"50f68e7a-3d6a-4da2-9264-06fec2b550e6","Type":"ContainerStarted","Data":"ddcbf45b03a6d2281caa28c07de8c1ddbcea13d3148d999db68306d95eecec61"}
Jul 17 12:44:01.533132 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.533096    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl"
Jul 17 12:44:01.533276 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:44:01.533248    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-hxlqh\" is forbidden: User \"system:node:ip-10-0-134-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-119.ec2.internal' and this object" podUID="0be23306-8e45-437a-bd80-82aa5ad3aa70" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh"
Jul 17 12:44:01.535647 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:44:01.535403    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nd5tp\" is forbidden: User \"system:node:ip-10-0-134-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-119.ec2.internal' and this object" podUID="5b1a2a61-8e79-4cf1-94c2-70c7f52843fb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"
Jul 17 12:44:01.535738 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.535651    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7q6d" event={"ID":"4182c07e-dbcf-4223-8253-da988273a232","Type":"ContainerStarted","Data":"e7cb1a3d2dd48cee9c9e5f22dbd4d29b7ae363052795b4365ce6df3ae6875753"}
Jul 17 12:44:01.535738 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.535685    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7q6d" event={"ID":"4182c07e-dbcf-4223-8253-da988273a232","Type":"ContainerStarted","Data":"e7e42523d8b6553af24fa4c71061f159bec5f458f021cdb5614dccce67cf1a19"}
Jul 17 12:44:01.536114 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.535976    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7q6d"
Jul 17 12:44:01.538062 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.538047    2565 scope.go:117] "RemoveContainer" containerID="876d6f7a437231201f89cb09fcaf6cbd0f5758a6b58af28e9f339d9c302a9d8a"
Jul 17 12:44:01.538289 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:44:01.538266    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"876d6f7a437231201f89cb09fcaf6cbd0f5758a6b58af28e9f339d9c302a9d8a\": container with ID starting with 876d6f7a437231201f89cb09fcaf6cbd0f5758a6b58af28e9f339d9c302a9d8a not found: ID does not exist" containerID="876d6f7a437231201f89cb09fcaf6cbd0f5758a6b58af28e9f339d9c302a9d8a"
Jul 17 12:44:01.538348 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.538299    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"876d6f7a437231201f89cb09fcaf6cbd0f5758a6b58af28e9f339d9c302a9d8a"} err="failed to get container status \"876d6f7a437231201f89cb09fcaf6cbd0f5758a6b58af28e9f339d9c302a9d8a\": rpc error: code = NotFound desc = could not find container \"876d6f7a437231201f89cb09fcaf6cbd0f5758a6b58af28e9f339d9c302a9d8a\": container with ID starting with 876d6f7a437231201f89cb09fcaf6cbd0f5758a6b58af28e9f339d9c302a9d8a not found: ID does not exist"
Jul 17 12:44:01.538348 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.538321    2565 scope.go:117] "RemoveContainer" containerID="3449ff5ce6cac24f7ee1259befa8ad9f5c1d1ab8e855b7f134be28c59720ef76"
Jul 17 12:44:01.546572 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.546558    2565 scope.go:117] "RemoveContainer" containerID="3449ff5ce6cac24f7ee1259befa8ad9f5c1d1ab8e855b7f134be28c59720ef76"
Jul 17 12:44:01.546802 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:44:01.546786    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3449ff5ce6cac24f7ee1259befa8ad9f5c1d1ab8e855b7f134be28c59720ef76\": container with ID starting with 3449ff5ce6cac24f7ee1259befa8ad9f5c1d1ab8e855b7f134be28c59720ef76 not found: ID does not exist" containerID="3449ff5ce6cac24f7ee1259befa8ad9f5c1d1ab8e855b7f134be28c59720ef76"
Jul 17 12:44:01.546858 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.546808    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3449ff5ce6cac24f7ee1259befa8ad9f5c1d1ab8e855b7f134be28c59720ef76"} err="failed to get container status \"3449ff5ce6cac24f7ee1259befa8ad9f5c1d1ab8e855b7f134be28c59720ef76\": rpc error: code = NotFound desc = could not find container \"3449ff5ce6cac24f7ee1259befa8ad9f5c1d1ab8e855b7f134be28c59720ef76\": container with ID starting with 3449ff5ce6cac24f7ee1259befa8ad9f5c1d1ab8e855b7f134be28c59720ef76 not found: ID does not exist"
Jul 17 12:44:01.556128 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.556091    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-4pvhs" podStartSLOduration=2.529970864 podStartE2EDuration="23.556078555s" totalImagesPullingTime="21.026107691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:43:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:44:01.554181629 +0000 UTC m=+1285.685566289" watchObservedRunningTime="2026-07-17 12:44:01.556078555 +0000 UTC m=+1285.687463244"
Jul 17 12:44:01.577697 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.577652    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7" podStartSLOduration=12.577638765 podStartE2EDuration="12.577638765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:44:01.576170128 +0000 UTC m=+1285.707554793" watchObservedRunningTime="2026-07-17 12:44:01.577638765 +0000 UTC m=+1285.709023425"
Jul 17 12:44:01.578581 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:44:01.578558    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-hxlqh\" is forbidden: User \"system:node:ip-10-0-134-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-119.ec2.internal' and this object" podUID="0be23306-8e45-437a-bd80-82aa5ad3aa70" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hxlqh"
Jul 17 12:44:01.599511 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.599470    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7q6d" podStartSLOduration=12.599460384 podStartE2EDuration="12.599460384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:44:01.59828264 +0000 UTC m=+1285.729667301" watchObservedRunningTime="2026-07-17 12:44:01.599460384 +0000 UTC m=+1285.730845044"
Jul 17 12:44:01.617375 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:01.617335    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl" podStartSLOduration=12.617321322 podStartE2EDuration="12.617321322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:44:01.616013515 +0000 UTC m=+1285.747398194" watchObservedRunningTime="2026-07-17 12:44:01.617321322 +0000 UTC m=+1285.748705984"
Jul 17 12:44:01.618319 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:44:01.618294    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-nd5tp\" is forbidden: User \"system:node:ip-10-0-134-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-119.ec2.internal' and this object" podUID="5b1a2a61-8e79-4cf1-94c2-70c7f52843fb" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nd5tp"
Jul 17 12:44:02.403058 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:02.403026    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0be23306-8e45-437a-bd80-82aa5ad3aa70" path="/var/lib/kubelet/pods/0be23306-8e45-437a-bd80-82aa5ad3aa70/volumes"
Jul 17 12:44:02.403420 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:02.403347    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b1a2a61-8e79-4cf1-94c2-70c7f52843fb" path="/var/lib/kubelet/pods/5b1a2a61-8e79-4cf1-94c2-70c7f52843fb/volumes"
Jul 17 12:44:05.679581 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:05.679542    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:44:05.817512 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:05.817481    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:44:12.549487 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:12.549458    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl"
Jul 17 12:44:12.549959 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:12.549526    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7"
Jul 17 12:44:12.652612 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:12.652579    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7"]
Jul 17 12:44:12.652829 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:12.652806    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7" podUID="426e349a-ef6d-4ad9-afca-8aa6104b3f00" containerName="manager" containerID="cri-o://764c5e89938097b51b3ddce7e304e306fa9f1d0ac77fa108b50e5d87492670b7" gracePeriod=10
Jul 17 12:44:12.898696 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:12.898675    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7"
Jul 17 12:44:13.061234 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:13.061185    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj42h\" (UniqueName: \"kubernetes.io/projected/426e349a-ef6d-4ad9-afca-8aa6104b3f00-kube-api-access-nj42h\") pod \"426e349a-ef6d-4ad9-afca-8aa6104b3f00\" (UID: \"426e349a-ef6d-4ad9-afca-8aa6104b3f00\") "
Jul 17 12:44:13.061414 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:13.061276    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/426e349a-ef6d-4ad9-afca-8aa6104b3f00-extensions-socket-volume\") pod \"426e349a-ef6d-4ad9-afca-8aa6104b3f00\" (UID: \"426e349a-ef6d-4ad9-afca-8aa6104b3f00\") "
Jul 17 12:44:13.061656 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:13.061631    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426e349a-ef6d-4ad9-afca-8aa6104b3f00-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "426e349a-ef6d-4ad9-afca-8aa6104b3f00" (UID: "426e349a-ef6d-4ad9-afca-8aa6104b3f00"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:44:13.063417 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:13.063392    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426e349a-ef6d-4ad9-afca-8aa6104b3f00-kube-api-access-nj42h" (OuterVolumeSpecName: "kube-api-access-nj42h") pod "426e349a-ef6d-4ad9-afca-8aa6104b3f00" (UID: "426e349a-ef6d-4ad9-afca-8aa6104b3f00"). InnerVolumeSpecName "kube-api-access-nj42h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:44:13.162744 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:13.162713    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj42h\" (UniqueName: \"kubernetes.io/projected/426e349a-ef6d-4ad9-afca-8aa6104b3f00-kube-api-access-nj42h\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:44:13.162744 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:13.162739    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/426e349a-ef6d-4ad9-afca-8aa6104b3f00-extensions-socket-volume\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:44:13.555571 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:13.555539    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7q6d"
Jul 17 12:44:13.611186 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:13.611156    2565 generic.go:356] "Generic (PLEG): container finished" podID="426e349a-ef6d-4ad9-afca-8aa6104b3f00" containerID="764c5e89938097b51b3ddce7e304e306fa9f1d0ac77fa108b50e5d87492670b7" exitCode=0
Jul 17 12:44:13.611358 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:13.611197    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7"
Jul 17 12:44:13.611358 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:13.611207    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7" event={"ID":"426e349a-ef6d-4ad9-afca-8aa6104b3f00","Type":"ContainerDied","Data":"764c5e89938097b51b3ddce7e304e306fa9f1d0ac77fa108b50e5d87492670b7"}
Jul 17 12:44:13.611358 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:13.611263    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7" event={"ID":"426e349a-ef6d-4ad9-afca-8aa6104b3f00","Type":"ContainerDied","Data":"874f73e72e834ed085b37e27ee9877f23ec4a7eceb0adc31280e09dd31fca71f"}
Jul 17 12:44:13.611358 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:13.611276    2565 scope.go:117] "RemoveContainer" containerID="764c5e89938097b51b3ddce7e304e306fa9f1d0ac77fa108b50e5d87492670b7"
Jul 17 12:44:13.621517 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:13.621497    2565 scope.go:117] "RemoveContainer" containerID="764c5e89938097b51b3ddce7e304e306fa9f1d0ac77fa108b50e5d87492670b7"
Jul 17 12:44:13.621776 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:44:13.621758    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"764c5e89938097b51b3ddce7e304e306fa9f1d0ac77fa108b50e5d87492670b7\": container with ID starting with 764c5e89938097b51b3ddce7e304e306fa9f1d0ac77fa108b50e5d87492670b7 not found: ID does not exist" containerID="764c5e89938097b51b3ddce7e304e306fa9f1d0ac77fa108b50e5d87492670b7"
Jul 17 12:44:13.621842 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:13.621788    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"764c5e89938097b51b3ddce7e304e306fa9f1d0ac77fa108b50e5d87492670b7"} err="failed to get container status \"764c5e89938097b51b3ddce7e304e306fa9f1d0ac77fa108b50e5d87492670b7\": rpc error: code = NotFound desc = could not find container \"764c5e89938097b51b3ddce7e304e306fa9f1d0ac77fa108b50e5d87492670b7\": container with ID starting with 764c5e89938097b51b3ddce7e304e306fa9f1d0ac77fa108b50e5d87492670b7 not found: ID does not exist"
Jul 17 12:44:13.633844 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:13.633820    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7"]
Jul 17 12:44:13.637654 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:13.637636    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tqsr7"]
Jul 17 12:44:14.402773 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:14.402732    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="426e349a-ef6d-4ad9-afca-8aa6104b3f00" path="/var/lib/kubelet/pods/426e349a-ef6d-4ad9-afca-8aa6104b3f00/volumes"
Jul 17 12:44:29.060244 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:29.060207    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:44:29.400038 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:29.399932    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:44:31.364629 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:31.364587    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:44:31.368401 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:31.368374    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k6x5f"]
Jul 17 12:44:31.368748 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:31.368736    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="426e349a-ef6d-4ad9-afca-8aa6104b3f00" containerName="manager"
Jul 17 12:44:31.368748 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:31.368750    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="426e349a-ef6d-4ad9-afca-8aa6104b3f00" containerName="manager"
Jul 17 12:44:31.368849 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:31.368840    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="426e349a-ef6d-4ad9-afca-8aa6104b3f00" containerName="manager"
Jul 17 12:44:31.401706 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:31.401677    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k6x5f"]
Jul 17 12:44:31.401822 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:31.401763    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6x5f"
Jul 17 12:44:31.404603 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:31.404582    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 17 12:44:31.462311 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:31.462276    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k6x5f"]
Jul 17 12:44:31.466501 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:31.466461    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:44:31.517706 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:31.517678    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2757a8f7-1c9e-47ed-86a8-feca99ff9295-config-file\") pod \"limitador-limitador-64c94fd64f-k6x5f\" (UID: \"2757a8f7-1c9e-47ed-86a8-feca99ff9295\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k6x5f"
Jul 17 12:44:31.517816 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:31.517717    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh9c5\" (UniqueName: \"kubernetes.io/projected/2757a8f7-1c9e-47ed-86a8-feca99ff9295-kube-api-access-qh9c5\") pod \"limitador-limitador-64c94fd64f-k6x5f\" (UID: \"2757a8f7-1c9e-47ed-86a8-feca99ff9295\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k6x5f"
Jul 17 12:44:31.577590 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:31.577560    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:44:31.618540 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:31.618482    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2757a8f7-1c9e-47ed-86a8-feca99ff9295-config-file\") pod \"limitador-limitador-64c94fd64f-k6x5f\" (UID: \"2757a8f7-1c9e-47ed-86a8-feca99ff9295\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k6x5f"
Jul 17 12:44:31.618540 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:31.618515    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh9c5\" (UniqueName: \"kubernetes.io/projected/2757a8f7-1c9e-47ed-86a8-feca99ff9295-kube-api-access-qh9c5\") pod \"limitador-limitador-64c94fd64f-k6x5f\" (UID: \"2757a8f7-1c9e-47ed-86a8-feca99ff9295\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k6x5f"
Jul 17 12:44:31.619125 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:31.619108    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2757a8f7-1c9e-47ed-86a8-feca99ff9295-config-file\") pod \"limitador-limitador-64c94fd64f-k6x5f\" (UID: \"2757a8f7-1c9e-47ed-86a8-feca99ff9295\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k6x5f"
Jul 17 12:44:31.626720 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:31.626699    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh9c5\" (UniqueName: \"kubernetes.io/projected/2757a8f7-1c9e-47ed-86a8-feca99ff9295-kube-api-access-qh9c5\") pod \"limitador-limitador-64c94fd64f-k6x5f\" (UID: \"2757a8f7-1c9e-47ed-86a8-feca99ff9295\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k6x5f"
Jul 17 12:44:31.712090 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:31.712067    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6x5f"
Jul 17 12:44:31.851708 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:31.851683    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k6x5f"]
Jul 17 12:44:31.853432 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:44:31.853408    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2757a8f7_1c9e_47ed_86a8_feca99ff9295.slice/crio-948f40e905d6cfef83fe626ce102dfd980a41b81804cb28c0a8f19d79b422f62 WatchSource:0}: Error finding container 948f40e905d6cfef83fe626ce102dfd980a41b81804cb28c0a8f19d79b422f62: Status 404 returned error can't find the container with id 948f40e905d6cfef83fe626ce102dfd980a41b81804cb28c0a8f19d79b422f62
Jul 17 12:44:32.186194 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:32.186159    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:44:32.262347 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:32.262321    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:44:32.331848 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:32.331822    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-q4gxw"]
Jul 17 12:44:32.335421 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:32.335403    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q4gxw"
Jul 17 12:44:32.338581 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:32.338562    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vvgsb\""
Jul 17 12:44:32.338796 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:32.338778    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q4gxw"]
Jul 17 12:44:32.426423 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:32.426401    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6lqq\" (UniqueName: \"kubernetes.io/projected/c66ac4b8-1e80-415d-a729-d466063efadb-kube-api-access-k6lqq\") pod \"authorino-75d556bd77-q4gxw\" (UID: \"c66ac4b8-1e80-415d-a729-d466063efadb\") " pod="kuadrant-system/authorino-75d556bd77-q4gxw"
Jul 17 12:44:32.527237 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:32.527173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6lqq\" (UniqueName: \"kubernetes.io/projected/c66ac4b8-1e80-415d-a729-d466063efadb-kube-api-access-k6lqq\") pod \"authorino-75d556bd77-q4gxw\" (UID: \"c66ac4b8-1e80-415d-a729-d466063efadb\") " pod="kuadrant-system/authorino-75d556bd77-q4gxw"
Jul 17 12:44:32.536079 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:32.536051    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6lqq\" (UniqueName: \"kubernetes.io/projected/c66ac4b8-1e80-415d-a729-d466063efadb-kube-api-access-k6lqq\") pod \"authorino-75d556bd77-q4gxw\" (UID: \"c66ac4b8-1e80-415d-a729-d466063efadb\") " pod="kuadrant-system/authorino-75d556bd77-q4gxw"
Jul 17 12:44:32.647832 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:32.647804    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q4gxw"
Jul 17 12:44:32.723007 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:32.722881    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6x5f" event={"ID":"2757a8f7-1c9e-47ed-86a8-feca99ff9295","Type":"ContainerStarted","Data":"948f40e905d6cfef83fe626ce102dfd980a41b81804cb28c0a8f19d79b422f62"}
Jul 17 12:44:32.829456 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:32.828942    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q4gxw"]
Jul 17 12:44:32.834481 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:44:32.834453    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc66ac4b8_1e80_415d_a729_d466063efadb.slice/crio-cbfe1e588e6a78ed76104d5962d0f9c420586eaa4fce70d894d325782b1335f5 WatchSource:0}: Error finding container cbfe1e588e6a78ed76104d5962d0f9c420586eaa4fce70d894d325782b1335f5: Status 404 returned error can't find the container with id cbfe1e588e6a78ed76104d5962d0f9c420586eaa4fce70d894d325782b1335f5
Jul 17 12:44:33.730932 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:33.730888    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q4gxw" event={"ID":"c66ac4b8-1e80-415d-a729-d466063efadb","Type":"ContainerStarted","Data":"cbfe1e588e6a78ed76104d5962d0f9c420586eaa4fce70d894d325782b1335f5"}
Jul 17 12:44:35.747031 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:35.746969    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6x5f" event={"ID":"2757a8f7-1c9e-47ed-86a8-feca99ff9295","Type":"ContainerStarted","Data":"00fdb5e1a91f710f5ecf26631a41de4b934b76e375cb9ba68fbbb7b56bbcc1ca"}
Jul 17 12:44:35.747416 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:35.747192    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6x5f"
Jul 17 12:44:36.756317 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:36.756274    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q4gxw" event={"ID":"c66ac4b8-1e80-415d-a729-d466063efadb","Type":"ContainerStarted","Data":"084bdbe42aeb77f8949675f05779fb335afec55476383c83a47612a94436563c"}
Jul 17 12:44:36.772381 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:36.772333    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-q4gxw" podStartSLOduration=1.013462438 podStartE2EDuration="4.772317906s" totalImagesPullingTime="3.758855468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:44:36.77187215 +0000 UTC m=+1320.903256812" watchObservedRunningTime="2026-07-17 12:44:36.772317906 +0000 UTC m=+1320.903702588"
Jul 17 12:44:36.773839 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:36.773807    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6x5f" podStartSLOduration=2.869074357 podStartE2EDuration="5.773794862s" totalImagesPullingTime="2.904720505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:44:35.764103843 +0000 UTC m=+1319.895488524" watchObservedRunningTime="2026-07-17 12:44:36.773794862 +0000 UTC m=+1320.905179522"
Jul 17 12:44:46.757226 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:46.757196    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6x5f"
Jul 17 12:44:46.833641 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:46.833604    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k6x5f"]
Jul 17 12:44:46.833823 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:46.833802    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6x5f" podUID="2757a8f7-1c9e-47ed-86a8-feca99ff9295" containerName="limitador" containerID="cri-o://00fdb5e1a91f710f5ecf26631a41de4b934b76e375cb9ba68fbbb7b56bbcc1ca" gracePeriod=30
Jul 17 12:44:47.385824 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:47.385799    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6x5f"
Jul 17 12:44:47.457241 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:47.457157    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh9c5\" (UniqueName: \"kubernetes.io/projected/2757a8f7-1c9e-47ed-86a8-feca99ff9295-kube-api-access-qh9c5\") pod \"2757a8f7-1c9e-47ed-86a8-feca99ff9295\" (UID: \"2757a8f7-1c9e-47ed-86a8-feca99ff9295\") "
Jul 17 12:44:47.457241 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:47.457196    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2757a8f7-1c9e-47ed-86a8-feca99ff9295-config-file\") pod \"2757a8f7-1c9e-47ed-86a8-feca99ff9295\" (UID: \"2757a8f7-1c9e-47ed-86a8-feca99ff9295\") "
Jul 17 12:44:47.457576 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:47.457552    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2757a8f7-1c9e-47ed-86a8-feca99ff9295-config-file" (OuterVolumeSpecName: "config-file") pod "2757a8f7-1c9e-47ed-86a8-feca99ff9295" (UID: "2757a8f7-1c9e-47ed-86a8-feca99ff9295"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 12:44:47.459420 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:47.459391    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2757a8f7-1c9e-47ed-86a8-feca99ff9295-kube-api-access-qh9c5" (OuterVolumeSpecName: "kube-api-access-qh9c5") pod "2757a8f7-1c9e-47ed-86a8-feca99ff9295" (UID: "2757a8f7-1c9e-47ed-86a8-feca99ff9295"). InnerVolumeSpecName "kube-api-access-qh9c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:44:47.557992 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:47.557963    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh9c5\" (UniqueName: \"kubernetes.io/projected/2757a8f7-1c9e-47ed-86a8-feca99ff9295-kube-api-access-qh9c5\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:44:47.558157 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:47.558007    2565 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2757a8f7-1c9e-47ed-86a8-feca99ff9295-config-file\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:44:47.823283 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:47.823255    2565 generic.go:356] "Generic (PLEG): container finished" podID="2757a8f7-1c9e-47ed-86a8-feca99ff9295" containerID="00fdb5e1a91f710f5ecf26631a41de4b934b76e375cb9ba68fbbb7b56bbcc1ca" exitCode=0
Jul 17 12:44:47.823720 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:47.823296    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6x5f" event={"ID":"2757a8f7-1c9e-47ed-86a8-feca99ff9295","Type":"ContainerDied","Data":"00fdb5e1a91f710f5ecf26631a41de4b934b76e375cb9ba68fbbb7b56bbcc1ca"}
Jul 17 12:44:47.823720 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:47.823300    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6x5f"
Jul 17 12:44:47.823720 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:47.823332    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-k6x5f" event={"ID":"2757a8f7-1c9e-47ed-86a8-feca99ff9295","Type":"ContainerDied","Data":"948f40e905d6cfef83fe626ce102dfd980a41b81804cb28c0a8f19d79b422f62"}
Jul 17 12:44:47.823720 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:47.823353    2565 scope.go:117] "RemoveContainer" containerID="00fdb5e1a91f710f5ecf26631a41de4b934b76e375cb9ba68fbbb7b56bbcc1ca"
Jul 17 12:44:47.832510 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:47.832496    2565 scope.go:117] "RemoveContainer" containerID="00fdb5e1a91f710f5ecf26631a41de4b934b76e375cb9ba68fbbb7b56bbcc1ca"
Jul 17 12:44:47.832762 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:44:47.832744    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00fdb5e1a91f710f5ecf26631a41de4b934b76e375cb9ba68fbbb7b56bbcc1ca\": container with ID starting with 00fdb5e1a91f710f5ecf26631a41de4b934b76e375cb9ba68fbbb7b56bbcc1ca not found: ID does not exist" containerID="00fdb5e1a91f710f5ecf26631a41de4b934b76e375cb9ba68fbbb7b56bbcc1ca"
Jul 17 12:44:47.832801 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:47.832772    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00fdb5e1a91f710f5ecf26631a41de4b934b76e375cb9ba68fbbb7b56bbcc1ca"} err="failed to get container status \"00fdb5e1a91f710f5ecf26631a41de4b934b76e375cb9ba68fbbb7b56bbcc1ca\": rpc error: code = NotFound desc = could not find container \"00fdb5e1a91f710f5ecf26631a41de4b934b76e375cb9ba68fbbb7b56bbcc1ca\": container with ID starting with 00fdb5e1a91f710f5ecf26631a41de4b934b76e375cb9ba68fbbb7b56bbcc1ca not found: ID does not exist"
Jul 17 12:44:47.844997 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:47.844956    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k6x5f"]
Jul 17 12:44:47.847073 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:47.847056    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k6x5f"]
Jul 17 12:44:48.403224 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:48.403192    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2757a8f7-1c9e-47ed-86a8-feca99ff9295" path="/var/lib/kubelet/pods/2757a8f7-1c9e-47ed-86a8-feca99ff9295/volumes"
Jul 17 12:44:52.523792 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:52.523757    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:44:52.622388 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:52.622352    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:44:52.989395 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:52.989360    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8z89s"]
Jul 17 12:44:52.989752 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:52.989738    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2757a8f7-1c9e-47ed-86a8-feca99ff9295" containerName="limitador"
Jul 17 12:44:52.989752 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:52.989752    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2757a8f7-1c9e-47ed-86a8-feca99ff9295" containerName="limitador"
Jul 17 12:44:52.989876 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:52.989824    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2757a8f7-1c9e-47ed-86a8-feca99ff9295" containerName="limitador"
Jul 17 12:44:52.994188 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:52.994162    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8z89s"
Jul 17 12:44:52.996972 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:52.996949    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 17 12:44:52.997093 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:52.996956    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-fhsgq\""
Jul 17 12:44:52.997150 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:52.997132    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 12:44:52.998192 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:52.998173    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 12:44:53.004184 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:53.004164    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8z89s"]
Jul 17 12:44:53.043687 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:53.043657    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:44:53.103722 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:53.103680    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ee43e5a0-0e95-48d3-8972-4c6876457f50-data\") pod \"postgres-69fb4594cf-8z89s\" (UID: \"ee43e5a0-0e95-48d3-8972-4c6876457f50\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8z89s"
Jul 17 12:44:53.103896 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:53.103728    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68vxb\" (UniqueName: \"kubernetes.io/projected/ee43e5a0-0e95-48d3-8972-4c6876457f50-kube-api-access-68vxb\") pod \"postgres-69fb4594cf-8z89s\" (UID: \"ee43e5a0-0e95-48d3-8972-4c6876457f50\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8z89s"
Jul 17 12:44:53.204488 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:53.204450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ee43e5a0-0e95-48d3-8972-4c6876457f50-data\") pod \"postgres-69fb4594cf-8z89s\" (UID: \"ee43e5a0-0e95-48d3-8972-4c6876457f50\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8z89s"
Jul 17 12:44:53.204680 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:53.204498    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68vxb\" (UniqueName: \"kubernetes.io/projected/ee43e5a0-0e95-48d3-8972-4c6876457f50-kube-api-access-68vxb\") pod \"postgres-69fb4594cf-8z89s\" (UID: \"ee43e5a0-0e95-48d3-8972-4c6876457f50\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8z89s"
Jul 17 12:44:53.204822 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:53.204802    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ee43e5a0-0e95-48d3-8972-4c6876457f50-data\") pod \"postgres-69fb4594cf-8z89s\" (UID: \"ee43e5a0-0e95-48d3-8972-4c6876457f50\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8z89s"
Jul 17 12:44:53.214625 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:53.214597    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68vxb\" (UniqueName: \"kubernetes.io/projected/ee43e5a0-0e95-48d3-8972-4c6876457f50-kube-api-access-68vxb\") pod \"postgres-69fb4594cf-8z89s\" (UID: \"ee43e5a0-0e95-48d3-8972-4c6876457f50\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-8z89s"
Jul 17 12:44:53.304954 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:53.304930    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8z89s"
Jul 17 12:44:53.444591 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:53.444570    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-8z89s"]
Jul 17 12:44:53.446236 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:44:53.446210    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee43e5a0_0e95_48d3_8972_4c6876457f50.slice/crio-b7fdb9ed161bd175e6dca7de6716118e63510fff1c3e497250ad5961f7715e97 WatchSource:0}: Error finding container b7fdb9ed161bd175e6dca7de6716118e63510fff1c3e497250ad5961f7715e97: Status 404 returned error can't find the container with id b7fdb9ed161bd175e6dca7de6716118e63510fff1c3e497250ad5961f7715e97
Jul 17 12:44:53.859740 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:53.859707    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8z89s" event={"ID":"ee43e5a0-0e95-48d3-8972-4c6876457f50","Type":"ContainerStarted","Data":"b7fdb9ed161bd175e6dca7de6716118e63510fff1c3e497250ad5961f7715e97"}
Jul 17 12:44:58.893058 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:58.893021    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8z89s" event={"ID":"ee43e5a0-0e95-48d3-8972-4c6876457f50","Type":"ContainerStarted","Data":"461dfecd75f883269c374303e18b6c838d62d2e7ef463887b1fe9e8b2f86f42a"}
Jul 17 12:44:58.893427 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:58.893088    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8z89s"
Jul 17 12:44:58.911442 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:44:58.911400    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8z89s" podStartSLOduration=2.293707185 podStartE2EDuration="6.911384642s" totalImagesPullingTime="4.617677457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:44:58.909267791 +0000 UTC m=+1343.040652478" watchObservedRunningTime="2026-07-17 12:44:58.911384642 +0000 UTC m=+1343.042769302"
Jul 17 12:45:04.888687 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:04.888625    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:45:04.929707 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:04.929682    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-8z89s"
Jul 17 12:45:04.988582 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:04.988552    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:45:05.253358 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:05.253271    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:45:05.530211 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:05.530132    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:45:05.734909 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:05.734880    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-848cd4747c-rx7ch"]
Jul 17 12:45:05.738738 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:05.738716    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-848cd4747c-rx7ch"
Jul 17 12:45:05.744917 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:05.744893    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-848cd4747c-rx7ch"]
Jul 17 12:45:05.905319 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:05.905291    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t75k4\" (UniqueName: \"kubernetes.io/projected/01fa8bfe-25e6-44bf-8083-61bd2e71d1ca-kube-api-access-t75k4\") pod \"authorino-848cd4747c-rx7ch\" (UID: \"01fa8bfe-25e6-44bf-8083-61bd2e71d1ca\") " pod="kuadrant-system/authorino-848cd4747c-rx7ch"
Jul 17 12:45:05.913872 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:05.913851    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-848cd4747c-rx7ch"]
Jul 17 12:45:05.914074 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:45:05.914057    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-t75k4], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-848cd4747c-rx7ch" podUID="01fa8bfe-25e6-44bf-8083-61bd2e71d1ca"
Jul 17 12:45:05.929923 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:05.929900    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-848cd4747c-rx7ch"
Jul 17 12:45:05.936511 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:05.936489    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-848cd4747c-rx7ch"
Jul 17 12:45:06.005957 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:06.005936    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t75k4\" (UniqueName: \"kubernetes.io/projected/01fa8bfe-25e6-44bf-8083-61bd2e71d1ca-kube-api-access-t75k4\") pod \"authorino-848cd4747c-rx7ch\" (UID: \"01fa8bfe-25e6-44bf-8083-61bd2e71d1ca\") " pod="kuadrant-system/authorino-848cd4747c-rx7ch"
Jul 17 12:45:06.014905 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:06.014887    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t75k4\" (UniqueName: \"kubernetes.io/projected/01fa8bfe-25e6-44bf-8083-61bd2e71d1ca-kube-api-access-t75k4\") pod \"authorino-848cd4747c-rx7ch\" (UID: \"01fa8bfe-25e6-44bf-8083-61bd2e71d1ca\") " pod="kuadrant-system/authorino-848cd4747c-rx7ch"
Jul 17 12:45:06.207092 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:06.207032    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t75k4\" (UniqueName: \"kubernetes.io/projected/01fa8bfe-25e6-44bf-8083-61bd2e71d1ca-kube-api-access-t75k4\") pod \"01fa8bfe-25e6-44bf-8083-61bd2e71d1ca\" (UID: \"01fa8bfe-25e6-44bf-8083-61bd2e71d1ca\") "
Jul 17 12:45:06.209048 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:06.209023    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fa8bfe-25e6-44bf-8083-61bd2e71d1ca-kube-api-access-t75k4" (OuterVolumeSpecName: "kube-api-access-t75k4") pod "01fa8bfe-25e6-44bf-8083-61bd2e71d1ca" (UID: "01fa8bfe-25e6-44bf-8083-61bd2e71d1ca"). InnerVolumeSpecName "kube-api-access-t75k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:45:06.308525 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:06.308500    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t75k4\" (UniqueName: \"kubernetes.io/projected/01fa8bfe-25e6-44bf-8083-61bd2e71d1ca-kube-api-access-t75k4\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:45:06.935129 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:06.935097    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-848cd4747c-rx7ch"
Jul 17 12:45:06.962711 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:06.962686    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-848cd4747c-rx7ch"]
Jul 17 12:45:06.966620 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:06.966598    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-848cd4747c-rx7ch"]
Jul 17 12:45:07.510540 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:07.510507    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q4gxw"]
Jul 17 12:45:07.510698 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:07.510678    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-q4gxw" podUID="c66ac4b8-1e80-415d-a729-d466063efadb" containerName="authorino" containerID="cri-o://084bdbe42aeb77f8949675f05779fb335afec55476383c83a47612a94436563c" gracePeriod=30
Jul 17 12:45:07.802601 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:07.802573    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q4gxw"
Jul 17 12:45:07.923904 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:07.923880    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6lqq\" (UniqueName: \"kubernetes.io/projected/c66ac4b8-1e80-415d-a729-d466063efadb-kube-api-access-k6lqq\") pod \"c66ac4b8-1e80-415d-a729-d466063efadb\" (UID: \"c66ac4b8-1e80-415d-a729-d466063efadb\") "
Jul 17 12:45:07.926152 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:07.926120    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66ac4b8-1e80-415d-a729-d466063efadb-kube-api-access-k6lqq" (OuterVolumeSpecName: "kube-api-access-k6lqq") pod "c66ac4b8-1e80-415d-a729-d466063efadb" (UID: "c66ac4b8-1e80-415d-a729-d466063efadb"). InnerVolumeSpecName "kube-api-access-k6lqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:45:07.941281 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:07.941252    2565 generic.go:356] "Generic (PLEG): container finished" podID="c66ac4b8-1e80-415d-a729-d466063efadb" containerID="084bdbe42aeb77f8949675f05779fb335afec55476383c83a47612a94436563c" exitCode=0
Jul 17 12:45:07.941627 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:07.941291    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-q4gxw"
Jul 17 12:45:07.941627 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:07.941325    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q4gxw" event={"ID":"c66ac4b8-1e80-415d-a729-d466063efadb","Type":"ContainerDied","Data":"084bdbe42aeb77f8949675f05779fb335afec55476383c83a47612a94436563c"}
Jul 17 12:45:07.941627 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:07.941361    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-q4gxw" event={"ID":"c66ac4b8-1e80-415d-a729-d466063efadb","Type":"ContainerDied","Data":"cbfe1e588e6a78ed76104d5962d0f9c420586eaa4fce70d894d325782b1335f5"}
Jul 17 12:45:07.941627 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:07.941378    2565 scope.go:117] "RemoveContainer" containerID="084bdbe42aeb77f8949675f05779fb335afec55476383c83a47612a94436563c"
Jul 17 12:45:07.951027 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:07.951008    2565 scope.go:117] "RemoveContainer" containerID="084bdbe42aeb77f8949675f05779fb335afec55476383c83a47612a94436563c"
Jul 17 12:45:07.951275 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:45:07.951258    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"084bdbe42aeb77f8949675f05779fb335afec55476383c83a47612a94436563c\": container with ID starting with 084bdbe42aeb77f8949675f05779fb335afec55476383c83a47612a94436563c not found: ID does not exist" containerID="084bdbe42aeb77f8949675f05779fb335afec55476383c83a47612a94436563c"
Jul 17 12:45:07.951328 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:07.951284    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"084bdbe42aeb77f8949675f05779fb335afec55476383c83a47612a94436563c"} err="failed to get container status \"084bdbe42aeb77f8949675f05779fb335afec55476383c83a47612a94436563c\": rpc error: code = NotFound desc = could not find container \"084bdbe42aeb77f8949675f05779fb335afec55476383c83a47612a94436563c\": container with ID starting with 084bdbe42aeb77f8949675f05779fb335afec55476383c83a47612a94436563c not found: ID does not exist"
Jul 17 12:45:07.964630 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:07.964607    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q4gxw"]
Jul 17 12:45:07.968287 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:07.968270    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-q4gxw"]
Jul 17 12:45:08.024911 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:08.024865    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6lqq\" (UniqueName: \"kubernetes.io/projected/c66ac4b8-1e80-415d-a729-d466063efadb-kube-api-access-k6lqq\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:45:08.403107 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:08.403077    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01fa8bfe-25e6-44bf-8083-61bd2e71d1ca" path="/var/lib/kubelet/pods/01fa8bfe-25e6-44bf-8083-61bd2e71d1ca/volumes"
Jul 17 12:45:08.403304 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:08.403292    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c66ac4b8-1e80-415d-a729-d466063efadb" path="/var/lib/kubelet/pods/c66ac4b8-1e80-415d-a729-d466063efadb/volumes"
Jul 17 12:45:12.262329 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.262293    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:45:12.273773 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.273731    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c6799484b-vtv4d"]
Jul 17 12:45:12.274204 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.274187    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c66ac4b8-1e80-415d-a729-d466063efadb" containerName="authorino"
Jul 17 12:45:12.274290 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.274206    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c66ac4b8-1e80-415d-a729-d466063efadb" containerName="authorino"
Jul 17 12:45:12.274347 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.274306    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c66ac4b8-1e80-415d-a729-d466063efadb" containerName="authorino"
Jul 17 12:45:12.289297 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.289275    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c6799484b-vtv4d"]
Jul 17 12:45:12.289420 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.289373    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c6799484b-vtv4d"
Jul 17 12:45:12.292331 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.292298    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 17 12:45:12.292331 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.292320    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2bz89\""
Jul 17 12:45:12.292502 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.292366    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 17 12:45:12.351953 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.351913    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b81323a-d89c-480f-a03a-c090781e53c8-webhook-cert\") pod \"maas-controller-5c6799484b-vtv4d\" (UID: \"1b81323a-d89c-480f-a03a-c090781e53c8\") " pod="opendatahub/maas-controller-5c6799484b-vtv4d"
Jul 17 12:45:12.352108 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.352022    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4rmg\" (UniqueName: \"kubernetes.io/projected/1b81323a-d89c-480f-a03a-c090781e53c8-kube-api-access-j4rmg\") pod \"maas-controller-5c6799484b-vtv4d\" (UID: \"1b81323a-d89c-480f-a03a-c090781e53c8\") " pod="opendatahub/maas-controller-5c6799484b-vtv4d"
Jul 17 12:45:12.362253 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.362233    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:45:12.415827 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.415806    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:45:12.452666 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.452643    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b81323a-d89c-480f-a03a-c090781e53c8-webhook-cert\") pod \"maas-controller-5c6799484b-vtv4d\" (UID: \"1b81323a-d89c-480f-a03a-c090781e53c8\") " pod="opendatahub/maas-controller-5c6799484b-vtv4d"
Jul 17 12:45:12.452783 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.452688    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4rmg\" (UniqueName: \"kubernetes.io/projected/1b81323a-d89c-480f-a03a-c090781e53c8-kube-api-access-j4rmg\") pod \"maas-controller-5c6799484b-vtv4d\" (UID: \"1b81323a-d89c-480f-a03a-c090781e53c8\") " pod="opendatahub/maas-controller-5c6799484b-vtv4d"
Jul 17 12:45:12.452783 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:45:12.452762    2565 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 17 12:45:12.452886 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:45:12.452836    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b81323a-d89c-480f-a03a-c090781e53c8-webhook-cert podName:1b81323a-d89c-480f-a03a-c090781e53c8 nodeName:}" failed. No retries permitted until 2026-07-17 12:45:12.952815383 +0000 UTC m=+1357.084200026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/1b81323a-d89c-480f-a03a-c090781e53c8-webhook-cert") pod "maas-controller-5c6799484b-vtv4d" (UID: "1b81323a-d89c-480f-a03a-c090781e53c8") : secret "maas-controller-webhook-cert" not found
Jul 17 12:45:12.456680 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.456660    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-85456b7f49-84lhh"]
Jul 17 12:45:12.459278 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.459261    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85456b7f49-84lhh"
Jul 17 12:45:12.463879 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.463857    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4rmg\" (UniqueName: \"kubernetes.io/projected/1b81323a-d89c-480f-a03a-c090781e53c8-kube-api-access-j4rmg\") pod \"maas-controller-5c6799484b-vtv4d\" (UID: \"1b81323a-d89c-480f-a03a-c090781e53c8\") " pod="opendatahub/maas-controller-5c6799484b-vtv4d"
Jul 17 12:45:12.469456 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.469434    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85456b7f49-84lhh"]
Jul 17 12:45:12.503836 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.503813    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:45:12.553604 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.553586    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq9ph\" (UniqueName: \"kubernetes.io/projected/fc8d3096-a73a-4d3b-8836-745f11003ab7-kube-api-access-dq9ph\") pod \"maas-controller-85456b7f49-84lhh\" (UID: \"fc8d3096-a73a-4d3b-8836-745f11003ab7\") " pod="opendatahub/maas-controller-85456b7f49-84lhh"
Jul 17 12:45:12.553705 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.553688    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc8d3096-a73a-4d3b-8836-745f11003ab7-webhook-cert\") pod \"maas-controller-85456b7f49-84lhh\" (UID: \"fc8d3096-a73a-4d3b-8836-745f11003ab7\") " pod="opendatahub/maas-controller-85456b7f49-84lhh"
Jul 17 12:45:12.654914 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.654890    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq9ph\" (UniqueName: \"kubernetes.io/projected/fc8d3096-a73a-4d3b-8836-745f11003ab7-kube-api-access-dq9ph\") pod \"maas-controller-85456b7f49-84lhh\" (UID: \"fc8d3096-a73a-4d3b-8836-745f11003ab7\") " pod="opendatahub/maas-controller-85456b7f49-84lhh"
Jul 17 12:45:12.655085 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.655021    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc8d3096-a73a-4d3b-8836-745f11003ab7-webhook-cert\") pod \"maas-controller-85456b7f49-84lhh\" (UID: \"fc8d3096-a73a-4d3b-8836-745f11003ab7\") " pod="opendatahub/maas-controller-85456b7f49-84lhh"
Jul 17 12:45:12.657509 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.657488    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc8d3096-a73a-4d3b-8836-745f11003ab7-webhook-cert\") pod \"maas-controller-85456b7f49-84lhh\" (UID: \"fc8d3096-a73a-4d3b-8836-745f11003ab7\") " pod="opendatahub/maas-controller-85456b7f49-84lhh"
Jul 17 12:45:12.662771 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.662752    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq9ph\" (UniqueName: \"kubernetes.io/projected/fc8d3096-a73a-4d3b-8836-745f11003ab7-kube-api-access-dq9ph\") pod \"maas-controller-85456b7f49-84lhh\" (UID: \"fc8d3096-a73a-4d3b-8836-745f11003ab7\") " pod="opendatahub/maas-controller-85456b7f49-84lhh"
Jul 17 12:45:12.779752 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.779725    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85456b7f49-84lhh"
Jul 17 12:45:12.920661 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.920634    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85456b7f49-84lhh"]
Jul 17 12:45:12.921881 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:45:12.921850    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8d3096_a73a_4d3b_8836_745f11003ab7.slice/crio-c02579166843f0449acf2b4ceae5867f393f1483665bb06c16e2d89f1f40ead7 WatchSource:0}: Error finding container c02579166843f0449acf2b4ceae5867f393f1483665bb06c16e2d89f1f40ead7: Status 404 returned error can't find the container with id c02579166843f0449acf2b4ceae5867f393f1483665bb06c16e2d89f1f40ead7
Jul 17 12:45:12.957273 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.957248    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b81323a-d89c-480f-a03a-c090781e53c8-webhook-cert\") pod \"maas-controller-5c6799484b-vtv4d\" (UID: \"1b81323a-d89c-480f-a03a-c090781e53c8\") " pod="opendatahub/maas-controller-5c6799484b-vtv4d"
Jul 17 12:45:12.959709 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.959691    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b81323a-d89c-480f-a03a-c090781e53c8-webhook-cert\") pod \"maas-controller-5c6799484b-vtv4d\" (UID: \"1b81323a-d89c-480f-a03a-c090781e53c8\") " pod="opendatahub/maas-controller-5c6799484b-vtv4d"
Jul 17 12:45:12.970650 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:12.970624    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85456b7f49-84lhh" event={"ID":"fc8d3096-a73a-4d3b-8836-745f11003ab7","Type":"ContainerStarted","Data":"c02579166843f0449acf2b4ceae5867f393f1483665bb06c16e2d89f1f40ead7"}
Jul 17 12:45:13.200438 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:13.200344    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c6799484b-vtv4d"
Jul 17 12:45:13.336901 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:13.336869    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c6799484b-vtv4d"]
Jul 17 12:45:13.338467 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:45:13.338440    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b81323a_d89c_480f_a03a_c090781e53c8.slice/crio-6aee022d403b7dcf5647622f56125b3c9b726ffdddf2f93266d49c6bf0451cd2 WatchSource:0}: Error finding container 6aee022d403b7dcf5647622f56125b3c9b726ffdddf2f93266d49c6bf0451cd2: Status 404 returned error can't find the container with id 6aee022d403b7dcf5647622f56125b3c9b726ffdddf2f93266d49c6bf0451cd2
Jul 17 12:45:13.340071 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:13.340053    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 12:45:13.982171 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:13.982090    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c6799484b-vtv4d" event={"ID":"1b81323a-d89c-480f-a03a-c090781e53c8","Type":"ContainerStarted","Data":"6aee022d403b7dcf5647622f56125b3c9b726ffdddf2f93266d49c6bf0451cd2"}
Jul 17 12:45:15.790194 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:15.790154    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:45:15.998195 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:15.998085    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85456b7f49-84lhh" event={"ID":"fc8d3096-a73a-4d3b-8836-745f11003ab7","Type":"ContainerStarted","Data":"b3e97e498fd09e84722c4aa45d7773083c34b772c97fef95a2c1be086990ab5b"}
Jul 17 12:45:15.998452 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:15.998424    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-85456b7f49-84lhh"
Jul 17 12:45:16.000047 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:16.000020    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c6799484b-vtv4d" event={"ID":"1b81323a-d89c-480f-a03a-c090781e53c8","Type":"ContainerStarted","Data":"a8a79accbcbdeeee7adef9d88b2be34d8087d6485b7310eceb8420b9d27a358c"}
Jul 17 12:45:16.000163 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:16.000079    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c6799484b-vtv4d"
Jul 17 12:45:16.019201 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:16.019122    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-85456b7f49-84lhh" podStartSLOduration=1.387285707 podStartE2EDuration="4.01910287s" totalImagesPullingTime="2.631817163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:45:16.016269917 +0000 UTC m=+1360.147654626" watchObservedRunningTime="2026-07-17 12:45:16.01910287 +0000 UTC m=+1360.150487529"
Jul 17 12:45:16.034685 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:16.034629    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5c6799484b-vtv4d" podStartSLOduration=1.809397937 podStartE2EDuration="4.034609448s" totalImagesPullingTime="2.225211511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:45:16.032916332 +0000 UTC m=+1360.164301025" watchObservedRunningTime="2026-07-17 12:45:16.034609448 +0000 UTC m=+1360.165994130"
Jul 17 12:45:16.547296 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:16.547258    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:45:27.008029 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:27.007976    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-85456b7f49-84lhh"
Jul 17 12:45:27.008456 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:27.008106    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c6799484b-vtv4d"
Jul 17 12:45:27.073867 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:27.073834    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c6799484b-vtv4d"]
Jul 17 12:45:27.074053 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:27.074033    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5c6799484b-vtv4d" podUID="1b81323a-d89c-480f-a03a-c090781e53c8" containerName="manager" containerID="cri-o://a8a79accbcbdeeee7adef9d88b2be34d8087d6485b7310eceb8420b9d27a358c" gracePeriod=30
Jul 17 12:45:27.322964 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:27.322943    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c6799484b-vtv4d"
Jul 17 12:45:27.364900 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:27.364868    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b81323a-d89c-480f-a03a-c090781e53c8-webhook-cert\") pod \"1b81323a-d89c-480f-a03a-c090781e53c8\" (UID: \"1b81323a-d89c-480f-a03a-c090781e53c8\") "
Jul 17 12:45:27.365090 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:27.364923    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4rmg\" (UniqueName: \"kubernetes.io/projected/1b81323a-d89c-480f-a03a-c090781e53c8-kube-api-access-j4rmg\") pod \"1b81323a-d89c-480f-a03a-c090781e53c8\" (UID: \"1b81323a-d89c-480f-a03a-c090781e53c8\") "
Jul 17 12:45:27.367316 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:27.367293    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b81323a-d89c-480f-a03a-c090781e53c8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "1b81323a-d89c-480f-a03a-c090781e53c8" (UID: "1b81323a-d89c-480f-a03a-c090781e53c8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:45:27.367386 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:27.367357    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b81323a-d89c-480f-a03a-c090781e53c8-kube-api-access-j4rmg" (OuterVolumeSpecName: "kube-api-access-j4rmg") pod "1b81323a-d89c-480f-a03a-c090781e53c8" (UID: "1b81323a-d89c-480f-a03a-c090781e53c8"). InnerVolumeSpecName "kube-api-access-j4rmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:45:27.466448 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:27.466412    2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b81323a-d89c-480f-a03a-c090781e53c8-webhook-cert\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:45:27.466448 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:27.466441    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4rmg\" (UniqueName: \"kubernetes.io/projected/1b81323a-d89c-480f-a03a-c090781e53c8-kube-api-access-j4rmg\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:45:28.069087 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:28.069051    2565 generic.go:356] "Generic (PLEG): container finished" podID="1b81323a-d89c-480f-a03a-c090781e53c8" containerID="a8a79accbcbdeeee7adef9d88b2be34d8087d6485b7310eceb8420b9d27a358c" exitCode=0
Jul 17 12:45:28.069539 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:28.069096    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c6799484b-vtv4d"
Jul 17 12:45:28.069539 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:28.069142    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c6799484b-vtv4d" event={"ID":"1b81323a-d89c-480f-a03a-c090781e53c8","Type":"ContainerDied","Data":"a8a79accbcbdeeee7adef9d88b2be34d8087d6485b7310eceb8420b9d27a358c"}
Jul 17 12:45:28.069539 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:28.069198    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c6799484b-vtv4d" event={"ID":"1b81323a-d89c-480f-a03a-c090781e53c8","Type":"ContainerDied","Data":"6aee022d403b7dcf5647622f56125b3c9b726ffdddf2f93266d49c6bf0451cd2"}
Jul 17 12:45:28.069539 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:28.069220    2565 scope.go:117] "RemoveContainer" containerID="a8a79accbcbdeeee7adef9d88b2be34d8087d6485b7310eceb8420b9d27a358c"
Jul 17 12:45:28.078966 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:28.078793    2565 scope.go:117] "RemoveContainer" containerID="a8a79accbcbdeeee7adef9d88b2be34d8087d6485b7310eceb8420b9d27a358c"
Jul 17 12:45:28.079119 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:45:28.079103    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8a79accbcbdeeee7adef9d88b2be34d8087d6485b7310eceb8420b9d27a358c\": container with ID starting with a8a79accbcbdeeee7adef9d88b2be34d8087d6485b7310eceb8420b9d27a358c not found: ID does not exist" containerID="a8a79accbcbdeeee7adef9d88b2be34d8087d6485b7310eceb8420b9d27a358c"
Jul 17 12:45:28.079167 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:28.079128    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a79accbcbdeeee7adef9d88b2be34d8087d6485b7310eceb8420b9d27a358c"} err="failed to get container status \"a8a79accbcbdeeee7adef9d88b2be34d8087d6485b7310eceb8420b9d27a358c\": rpc error: code = NotFound desc = could not find container \"a8a79accbcbdeeee7adef9d88b2be34d8087d6485b7310eceb8420b9d27a358c\": container with ID starting with a8a79accbcbdeeee7adef9d88b2be34d8087d6485b7310eceb8420b9d27a358c not found: ID does not exist"
Jul 17 12:45:28.090755 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:28.090732    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c6799484b-vtv4d"]
Jul 17 12:45:28.095810 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:28.095790    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5c6799484b-vtv4d"]
Jul 17 12:45:28.403517 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:28.403433    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b81323a-d89c-480f-a03a-c090781e53c8" path="/var/lib/kubelet/pods/1b81323a-d89c-480f-a03a-c090781e53c8/volumes"
Jul 17 12:45:47.547794 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:47.547762    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:45:47.712418 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:47.712383    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:45:48.045457 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.045361    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:45:48.197857 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.197825    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5cbc55679f-gw844"]
Jul 17 12:45:48.198290 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.198272    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b81323a-d89c-480f-a03a-c090781e53c8" containerName="manager"
Jul 17 12:45:48.198351 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.198293    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b81323a-d89c-480f-a03a-c090781e53c8" containerName="manager"
Jul 17 12:45:48.198423 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.198412    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b81323a-d89c-480f-a03a-c090781e53c8" containerName="manager"
Jul 17 12:45:48.203791 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.203772    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cbc55679f-gw844"
Jul 17 12:45:48.208613 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.208594    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lwvzj\""
Jul 17 12:45:48.209899 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.209882    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 17 12:45:48.222837 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.222814    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cbc55679f-gw844"]
Jul 17 12:45:48.282321 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.282290    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:45:48.344414 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.344372    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxc9t\" (UniqueName: \"kubernetes.io/projected/bcd7660c-d7f1-4d9d-81bf-c5c45ae9a660-kube-api-access-wxc9t\") pod \"maas-api-5cbc55679f-gw844\" (UID: \"bcd7660c-d7f1-4d9d-81bf-c5c45ae9a660\") " pod="odh-ai-gateway-infra/maas-api-5cbc55679f-gw844"
Jul 17 12:45:48.344568 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.344496    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcd7660c-d7f1-4d9d-81bf-c5c45ae9a660-maas-api-tls\") pod \"maas-api-5cbc55679f-gw844\" (UID: \"bcd7660c-d7f1-4d9d-81bf-c5c45ae9a660\") " pod="odh-ai-gateway-infra/maas-api-5cbc55679f-gw844"
Jul 17 12:45:48.344568 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.344519    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd7660c-d7f1-4d9d-81bf-c5c45ae9a660-openshift-service-ca\") pod \"maas-api-5cbc55679f-gw844\" (UID: \"bcd7660c-d7f1-4d9d-81bf-c5c45ae9a660\") " pod="odh-ai-gateway-infra/maas-api-5cbc55679f-gw844"
Jul 17 12:45:48.445426 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.445399    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcd7660c-d7f1-4d9d-81bf-c5c45ae9a660-maas-api-tls\") pod \"maas-api-5cbc55679f-gw844\" (UID: \"bcd7660c-d7f1-4d9d-81bf-c5c45ae9a660\") " pod="odh-ai-gateway-infra/maas-api-5cbc55679f-gw844"
Jul 17 12:45:48.445596 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.445434    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd7660c-d7f1-4d9d-81bf-c5c45ae9a660-openshift-service-ca\") pod \"maas-api-5cbc55679f-gw844\" (UID: \"bcd7660c-d7f1-4d9d-81bf-c5c45ae9a660\") " pod="odh-ai-gateway-infra/maas-api-5cbc55679f-gw844"
Jul 17 12:45:48.445596 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.445467    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxc9t\" (UniqueName: \"kubernetes.io/projected/bcd7660c-d7f1-4d9d-81bf-c5c45ae9a660-kube-api-access-wxc9t\") pod \"maas-api-5cbc55679f-gw844\" (UID: \"bcd7660c-d7f1-4d9d-81bf-c5c45ae9a660\") " pod="odh-ai-gateway-infra/maas-api-5cbc55679f-gw844"
Jul 17 12:45:48.446135 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.446114    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd7660c-d7f1-4d9d-81bf-c5c45ae9a660-openshift-service-ca\") pod \"maas-api-5cbc55679f-gw844\" (UID: \"bcd7660c-d7f1-4d9d-81bf-c5c45ae9a660\") " pod="odh-ai-gateway-infra/maas-api-5cbc55679f-gw844"
Jul 17 12:45:48.448098 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.448081    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcd7660c-d7f1-4d9d-81bf-c5c45ae9a660-maas-api-tls\") pod \"maas-api-5cbc55679f-gw844\" (UID: \"bcd7660c-d7f1-4d9d-81bf-c5c45ae9a660\") " pod="odh-ai-gateway-infra/maas-api-5cbc55679f-gw844"
Jul 17 12:45:48.454635 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.454612    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxc9t\" (UniqueName: \"kubernetes.io/projected/bcd7660c-d7f1-4d9d-81bf-c5c45ae9a660-kube-api-access-wxc9t\") pod \"maas-api-5cbc55679f-gw844\" (UID: \"bcd7660c-d7f1-4d9d-81bf-c5c45ae9a660\") " pod="odh-ai-gateway-infra/maas-api-5cbc55679f-gw844"
Jul 17 12:45:48.514626 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.514594    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cbc55679f-gw844"
Jul 17 12:45:48.707909 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.707879    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cbc55679f-gw844"]
Jul 17 12:45:48.709317 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:45:48.709293    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd7660c_d7f1_4d9d_81bf_c5c45ae9a660.slice/crio-094a2e86f1be4b898ffa5a11e408c30e8dd93b68346d373c825f88502a8c5c51 WatchSource:0}: Error finding container 094a2e86f1be4b898ffa5a11e408c30e8dd93b68346d373c825f88502a8c5c51: Status 404 returned error can't find the container with id 094a2e86f1be4b898ffa5a11e408c30e8dd93b68346d373c825f88502a8c5c51
Jul 17 12:45:48.786527 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.786496    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:45:48.921002 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:48.920871    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:45:49.026735 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.026704    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-62qbl"]
Jul 17 12:45:49.030153 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.030137    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-62qbl"
Jul 17 12:45:49.033894 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.033870    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 17 12:45:49.034079 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.033937    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2vggr\""
Jul 17 12:45:49.047335 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.047310    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-62qbl"]
Jul 17 12:45:49.074603 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.074581    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-pbxkj"]
Jul 17 12:45:49.077207 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.077183    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-pbxkj"
Jul 17 12:45:49.104919 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.104895    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-pbxkj"]
Jul 17 12:45:49.152852 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.152820    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgs2b\" (UniqueName: \"kubernetes.io/projected/dfb10fec-52a0-4c07-a09b-5c4e6f7d52de-kube-api-access-pgs2b\") pod \"payload-pre-processing-7677b774bb-62qbl\" (UID: \"dfb10fec-52a0-4c07-a09b-5c4e6f7d52de\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-62qbl"
Jul 17 12:45:49.152977 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.152894    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798c57d9-278e-4438-b436-a1e2f3280a8e-plugins-config-volume\") pod \"payload-processing-5bf47cd988-pbxkj\" (UID: \"798c57d9-278e-4438-b436-a1e2f3280a8e\") " pod="openshift-ingress/payload-processing-5bf47cd988-pbxkj"
Jul 17 12:45:49.152977 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.152919    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb10fec-52a0-4c07-a09b-5c4e6f7d52de-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-62qbl\" (UID: \"dfb10fec-52a0-4c07-a09b-5c4e6f7d52de\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-62qbl"
Jul 17 12:45:49.152977 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.152969    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs2m6\" (UniqueName: \"kubernetes.io/projected/798c57d9-278e-4438-b436-a1e2f3280a8e-kube-api-access-fs2m6\") pod \"payload-processing-5bf47cd988-pbxkj\" (UID: \"798c57d9-278e-4438-b436-a1e2f3280a8e\") " pod="openshift-ingress/payload-processing-5bf47cd988-pbxkj"
Jul 17 12:45:49.197400 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.197290    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cbc55679f-gw844" event={"ID":"bcd7660c-d7f1-4d9d-81bf-c5c45ae9a660","Type":"ContainerStarted","Data":"094a2e86f1be4b898ffa5a11e408c30e8dd93b68346d373c825f88502a8c5c51"}
Jul 17 12:45:49.253887 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.253855    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgs2b\" (UniqueName: \"kubernetes.io/projected/dfb10fec-52a0-4c07-a09b-5c4e6f7d52de-kube-api-access-pgs2b\") pod \"payload-pre-processing-7677b774bb-62qbl\" (UID: \"dfb10fec-52a0-4c07-a09b-5c4e6f7d52de\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-62qbl"
Jul 17 12:45:49.254060 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.253896    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798c57d9-278e-4438-b436-a1e2f3280a8e-plugins-config-volume\") pod \"payload-processing-5bf47cd988-pbxkj\" (UID: \"798c57d9-278e-4438-b436-a1e2f3280a8e\") " pod="openshift-ingress/payload-processing-5bf47cd988-pbxkj"
Jul 17 12:45:49.254060 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.253916    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb10fec-52a0-4c07-a09b-5c4e6f7d52de-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-62qbl\" (UID: \"dfb10fec-52a0-4c07-a09b-5c4e6f7d52de\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-62qbl"
Jul 17 12:45:49.254224 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.254055    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs2m6\" (UniqueName: \"kubernetes.io/projected/798c57d9-278e-4438-b436-a1e2f3280a8e-kube-api-access-fs2m6\") pod \"payload-processing-5bf47cd988-pbxkj\" (UID: \"798c57d9-278e-4438-b436-a1e2f3280a8e\") " pod="openshift-ingress/payload-processing-5bf47cd988-pbxkj"
Jul 17 12:45:49.254836 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.254811    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb10fec-52a0-4c07-a09b-5c4e6f7d52de-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-62qbl\" (UID: \"dfb10fec-52a0-4c07-a09b-5c4e6f7d52de\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-62qbl"
Jul 17 12:45:49.254927 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.254818    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/798c57d9-278e-4438-b436-a1e2f3280a8e-plugins-config-volume\") pod \"payload-processing-5bf47cd988-pbxkj\" (UID: \"798c57d9-278e-4438-b436-a1e2f3280a8e\") " pod="openshift-ingress/payload-processing-5bf47cd988-pbxkj"
Jul 17 12:45:49.295208 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.295189    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs2m6\" (UniqueName: \"kubernetes.io/projected/798c57d9-278e-4438-b436-a1e2f3280a8e-kube-api-access-fs2m6\") pod \"payload-processing-5bf47cd988-pbxkj\" (UID: \"798c57d9-278e-4438-b436-a1e2f3280a8e\") " pod="openshift-ingress/payload-processing-5bf47cd988-pbxkj"
Jul 17 12:45:49.299062 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.299034    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgs2b\" (UniqueName: \"kubernetes.io/projected/dfb10fec-52a0-4c07-a09b-5c4e6f7d52de-kube-api-access-pgs2b\") pod \"payload-pre-processing-7677b774bb-62qbl\" (UID: \"dfb10fec-52a0-4c07-a09b-5c4e6f7d52de\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-62qbl"
Jul 17 12:45:49.341089 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.341057    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-62qbl"
Jul 17 12:45:49.387101 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.387070    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-pbxkj"
Jul 17 12:45:49.522926 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.522849    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-62qbl"]
Jul 17 12:45:49.523682 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:45:49.523650    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb10fec_52a0_4c07_a09b_5c4e6f7d52de.slice/crio-1ff935de4a2f6dacd1c58fb2f9b5150c7c5824064b4237ac2cde25febad7e50b WatchSource:0}: Error finding container 1ff935de4a2f6dacd1c58fb2f9b5150c7c5824064b4237ac2cde25febad7e50b: Status 404 returned error can't find the container with id 1ff935de4a2f6dacd1c58fb2f9b5150c7c5824064b4237ac2cde25febad7e50b
Jul 17 12:45:49.592375 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:49.592352    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-pbxkj"]
Jul 17 12:45:49.594879 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:45:49.594850    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798c57d9_278e_4438_b436_a1e2f3280a8e.slice/crio-6acc3142933672f4cca88a3df227b34f8d876cf021eaab2d7eeaa5ef3c3c10b6 WatchSource:0}: Error finding container 6acc3142933672f4cca88a3df227b34f8d876cf021eaab2d7eeaa5ef3c3c10b6: Status 404 returned error can't find the container with id 6acc3142933672f4cca88a3df227b34f8d876cf021eaab2d7eeaa5ef3c3c10b6
Jul 17 12:45:50.208018 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:50.207877    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-pbxkj" event={"ID":"798c57d9-278e-4438-b436-a1e2f3280a8e","Type":"ContainerStarted","Data":"6acc3142933672f4cca88a3df227b34f8d876cf021eaab2d7eeaa5ef3c3c10b6"}
Jul 17 12:45:50.210422 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:50.210364    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-62qbl" event={"ID":"dfb10fec-52a0-4c07-a09b-5c4e6f7d52de","Type":"ContainerStarted","Data":"1ff935de4a2f6dacd1c58fb2f9b5150c7c5824064b4237ac2cde25febad7e50b"}
Jul 17 12:45:51.220384 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:51.220304    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cbc55679f-gw844" event={"ID":"bcd7660c-d7f1-4d9d-81bf-c5c45ae9a660","Type":"ContainerStarted","Data":"e0f4936f98505b05e06d74bf980e7b4f89623f9cc4a44eb00062ab9be6cefb48"}
Jul 17 12:45:51.220814 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:51.220674    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5cbc55679f-gw844"
Jul 17 12:45:51.250161 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:51.250112    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5cbc55679f-gw844" podStartSLOduration=1.082597513 podStartE2EDuration="3.250091516s" totalImagesPullingTime="2.167494003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:45:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:45:51.249664611 +0000 UTC m=+1395.381049295" watchObservedRunningTime="2026-07-17 12:45:51.250091516 +0000 UTC m=+1395.381476203"
Jul 17 12:45:52.227241 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:52.227192    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-pbxkj" event={"ID":"798c57d9-278e-4438-b436-a1e2f3280a8e","Type":"ContainerStarted","Data":"1c032572ea7b924fdd0a7e438bb4677399312afb8ea44dd8d0ee6d5c954d559e"}
Jul 17 12:45:52.227687 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:52.227271    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-pbxkj"
Jul 17 12:45:52.228894 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:52.228869    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-62qbl" event={"ID":"dfb10fec-52a0-4c07-a09b-5c4e6f7d52de","Type":"ContainerStarted","Data":"98ba58de224522ca34520eb8439e5c41803c01b55f1ea39305ad1070f84cf54f"}
Jul 17 12:45:52.229148 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:52.229133    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-62qbl"
Jul 17 12:45:52.247733 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:52.247692    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-pbxkj" podStartSLOduration=1.4326497919999999 podStartE2EDuration="3.247678988s" totalImagesPullingTime="1.815029196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:45:52.246372226 +0000 UTC m=+1396.377756908" watchObservedRunningTime="2026-07-17 12:45:52.247678988 +0000 UTC m=+1396.379063627"
Jul 17 12:45:52.265637 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:52.265596    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-62qbl" podStartSLOduration=2.375029021 podStartE2EDuration="4.26558482s" totalImagesPullingTime="1.890555799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:45:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:45:52.263332656 +0000 UTC m=+1396.394717341" watchObservedRunningTime="2026-07-17 12:45:52.26558482 +0000 UTC m=+1396.396969480"
Jul 17 12:45:53.307195 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:53.307164    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:45:53.405858 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:53.405829    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:45:57.234447 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:57.234420    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5cbc55679f-gw844"
Jul 17 12:45:59.086690 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:59.086651    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:45:59.187218 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:45:59.187187    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:46:03.237222 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:03.237195    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-62qbl"
Jul 17 12:46:03.237599 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:03.237263    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-pbxkj"
Jul 17 12:46:11.383211 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.383178    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"]
Jul 17 12:46:11.386711 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.386692    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:11.389619 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.389598    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 17 12:46:11.389805 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.389787    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hdpj4\""
Jul 17 12:46:11.389885 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.389788    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 17 12:46:11.390855 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.390839    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 17 12:46:11.397065 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.397036    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"]
Jul 17 12:46:11.404817 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.404792    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:46:11.536413 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.536377    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94d7f364-bb60-42ee-b6a0-bd7535023b8e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt\" (UID: \"94d7f364-bb60-42ee-b6a0-bd7535023b8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:11.536581 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.536533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdxlz\" (UniqueName: \"kubernetes.io/projected/94d7f364-bb60-42ee-b6a0-bd7535023b8e-kube-api-access-xdxlz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt\" (UID: \"94d7f364-bb60-42ee-b6a0-bd7535023b8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:11.536581 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.536573    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d7f364-bb60-42ee-b6a0-bd7535023b8e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt\" (UID: \"94d7f364-bb60-42ee-b6a0-bd7535023b8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:11.536699 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.536646    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94d7f364-bb60-42ee-b6a0-bd7535023b8e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt\" (UID: \"94d7f364-bb60-42ee-b6a0-bd7535023b8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:11.536746 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.536699    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94d7f364-bb60-42ee-b6a0-bd7535023b8e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt\" (UID: \"94d7f364-bb60-42ee-b6a0-bd7535023b8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:11.536783 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.536772    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94d7f364-bb60-42ee-b6a0-bd7535023b8e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt\" (UID: \"94d7f364-bb60-42ee-b6a0-bd7535023b8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:11.637349 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.637279    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94d7f364-bb60-42ee-b6a0-bd7535023b8e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt\" (UID: \"94d7f364-bb60-42ee-b6a0-bd7535023b8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:11.637349 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.637320    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94d7f364-bb60-42ee-b6a0-bd7535023b8e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt\" (UID: \"94d7f364-bb60-42ee-b6a0-bd7535023b8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:11.637527 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.637353    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdxlz\" (UniqueName: \"kubernetes.io/projected/94d7f364-bb60-42ee-b6a0-bd7535023b8e-kube-api-access-xdxlz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt\" (UID: \"94d7f364-bb60-42ee-b6a0-bd7535023b8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:11.637527 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.637372    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d7f364-bb60-42ee-b6a0-bd7535023b8e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt\" (UID: \"94d7f364-bb60-42ee-b6a0-bd7535023b8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:11.637527 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.637403    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94d7f364-bb60-42ee-b6a0-bd7535023b8e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt\" (UID: \"94d7f364-bb60-42ee-b6a0-bd7535023b8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:11.637527 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.637423    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94d7f364-bb60-42ee-b6a0-bd7535023b8e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt\" (UID: \"94d7f364-bb60-42ee-b6a0-bd7535023b8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:11.637805 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.637778    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94d7f364-bb60-42ee-b6a0-bd7535023b8e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt\" (UID: \"94d7f364-bb60-42ee-b6a0-bd7535023b8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:11.637919 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.637827    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94d7f364-bb60-42ee-b6a0-bd7535023b8e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt\" (UID: \"94d7f364-bb60-42ee-b6a0-bd7535023b8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:11.637919 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.637844    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94d7f364-bb60-42ee-b6a0-bd7535023b8e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt\" (UID: \"94d7f364-bb60-42ee-b6a0-bd7535023b8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:11.639723 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.639697    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94d7f364-bb60-42ee-b6a0-bd7535023b8e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt\" (UID: \"94d7f364-bb60-42ee-b6a0-bd7535023b8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:11.639871 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.639856    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94d7f364-bb60-42ee-b6a0-bd7535023b8e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt\" (UID: \"94d7f364-bb60-42ee-b6a0-bd7535023b8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:11.647080 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.647056    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdxlz\" (UniqueName: \"kubernetes.io/projected/94d7f364-bb60-42ee-b6a0-bd7535023b8e-kube-api-access-xdxlz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt\" (UID: \"94d7f364-bb60-42ee-b6a0-bd7535023b8e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:11.698963 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.698940    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:11.842738 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:11.842707    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"]
Jul 17 12:46:11.843320 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:46:11.843289    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d7f364_bb60_42ee_b6a0_bd7535023b8e.slice/crio-15bd47a2df8e60e67e511872e645e2155bc7dd5f0e8f2977a69baa5803a7abcc WatchSource:0}: Error finding container 15bd47a2df8e60e67e511872e645e2155bc7dd5f0e8f2977a69baa5803a7abcc: Status 404 returned error can't find the container with id 15bd47a2df8e60e67e511872e645e2155bc7dd5f0e8f2977a69baa5803a7abcc
Jul 17 12:46:12.348335 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:12.348282    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt" event={"ID":"94d7f364-bb60-42ee-b6a0-bd7535023b8e","Type":"ContainerStarted","Data":"15bd47a2df8e60e67e511872e645e2155bc7dd5f0e8f2977a69baa5803a7abcc"}
Jul 17 12:46:13.355496 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:13.355408    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt" event={"ID":"94d7f364-bb60-42ee-b6a0-bd7535023b8e","Type":"ContainerStarted","Data":"926186381bd07651c290416d3756d2401022858594bf6b3adf00e019101ed8da"}
Jul 17 12:46:13.376361 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:13.376155    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt" podStartSLOduration=1.14577182 podStartE2EDuration="2.376138453s" totalImagesPullingTime="1.230366633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:46:13.375066607 +0000 UTC m=+1417.506451268" watchObservedRunningTime="2026-07-17 12:46:13.376138453 +0000 UTC m=+1417.507523114"
Jul 17 12:46:21.699648 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:21.699612    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:21.700048 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:21.699684    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:21.711766 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:21.711743    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:22.418091 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:22.418062    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt"
Jul 17 12:46:45.688657 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:45.688620    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:46:45.744417 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:45.744387    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:46:48.458118 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:48.458085    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:46:48.512557 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:48.512532    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:46:49.671842 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.671812    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"]
Jul 17 12:46:49.678922 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.678892    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:46:49.681599 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.681579    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 17 12:46:49.687963 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.687355    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"]
Jul 17 12:46:49.705493 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.705463    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:46:49.788742 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.788702    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7qtd\" (UniqueName: \"kubernetes.io/projected/1664b90e-7ff9-4901-ab0c-0df556f4f295-kube-api-access-m7qtd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-crf9d\" (UID: \"1664b90e-7ff9-4901-ab0c-0df556f4f295\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:46:49.788943 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.788831    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1664b90e-7ff9-4901-ab0c-0df556f4f295-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-crf9d\" (UID: \"1664b90e-7ff9-4901-ab0c-0df556f4f295\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:46:49.788943 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.788882    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1664b90e-7ff9-4901-ab0c-0df556f4f295-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-crf9d\" (UID: \"1664b90e-7ff9-4901-ab0c-0df556f4f295\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:46:49.788943 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.788917    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1664b90e-7ff9-4901-ab0c-0df556f4f295-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-crf9d\" (UID: \"1664b90e-7ff9-4901-ab0c-0df556f4f295\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:46:49.789162 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.789032    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1664b90e-7ff9-4901-ab0c-0df556f4f295-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-crf9d\" (UID: \"1664b90e-7ff9-4901-ab0c-0df556f4f295\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:46:49.789162 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.789068    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1664b90e-7ff9-4901-ab0c-0df556f4f295-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-crf9d\" (UID: \"1664b90e-7ff9-4901-ab0c-0df556f4f295\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:46:49.890347 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.890310    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1664b90e-7ff9-4901-ab0c-0df556f4f295-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-crf9d\" (UID: \"1664b90e-7ff9-4901-ab0c-0df556f4f295\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:46:49.890347 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.890347    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1664b90e-7ff9-4901-ab0c-0df556f4f295-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-crf9d\" (UID: \"1664b90e-7ff9-4901-ab0c-0df556f4f295\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:46:49.890565 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.890375    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7qtd\" (UniqueName: \"kubernetes.io/projected/1664b90e-7ff9-4901-ab0c-0df556f4f295-kube-api-access-m7qtd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-crf9d\" (UID: \"1664b90e-7ff9-4901-ab0c-0df556f4f295\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:46:49.890565 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.890445    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1664b90e-7ff9-4901-ab0c-0df556f4f295-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-crf9d\" (UID: \"1664b90e-7ff9-4901-ab0c-0df556f4f295\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:46:49.890565 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.890489    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1664b90e-7ff9-4901-ab0c-0df556f4f295-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-crf9d\" (UID: \"1664b90e-7ff9-4901-ab0c-0df556f4f295\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:46:49.890565 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.890517    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1664b90e-7ff9-4901-ab0c-0df556f4f295-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-crf9d\" (UID: \"1664b90e-7ff9-4901-ab0c-0df556f4f295\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:46:49.890861 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.890839    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1664b90e-7ff9-4901-ab0c-0df556f4f295-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-crf9d\" (UID: \"1664b90e-7ff9-4901-ab0c-0df556f4f295\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:46:49.891083 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.890885    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1664b90e-7ff9-4901-ab0c-0df556f4f295-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-crf9d\" (UID: \"1664b90e-7ff9-4901-ab0c-0df556f4f295\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:46:49.891083 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.890963    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1664b90e-7ff9-4901-ab0c-0df556f4f295-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-crf9d\" (UID: \"1664b90e-7ff9-4901-ab0c-0df556f4f295\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:46:49.892787 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.892765    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1664b90e-7ff9-4901-ab0c-0df556f4f295-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-crf9d\" (UID: \"1664b90e-7ff9-4901-ab0c-0df556f4f295\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:46:49.892959 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.892942    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1664b90e-7ff9-4901-ab0c-0df556f4f295-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-crf9d\" (UID: \"1664b90e-7ff9-4901-ab0c-0df556f4f295\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:46:49.898813 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.898787    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7qtd\" (UniqueName: \"kubernetes.io/projected/1664b90e-7ff9-4901-ab0c-0df556f4f295-kube-api-access-m7qtd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-crf9d\" (UID: \"1664b90e-7ff9-4901-ab0c-0df556f4f295\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:46:49.991834 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:49.991736    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:46:50.140321 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:50.140292    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"]
Jul 17 12:46:50.141045 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:46:50.141020    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1664b90e_7ff9_4901_ab0c_0df556f4f295.slice/crio-1b43fedd90863e2aa662262e78c9deac27e842eb9fd32d01d5655bb295bca056 WatchSource:0}: Error finding container 1b43fedd90863e2aa662262e78c9deac27e842eb9fd32d01d5655bb295bca056: Status 404 returned error can't find the container with id 1b43fedd90863e2aa662262e78c9deac27e842eb9fd32d01d5655bb295bca056
Jul 17 12:46:50.573628 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:50.573596    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d" event={"ID":"1664b90e-7ff9-4901-ab0c-0df556f4f295","Type":"ContainerStarted","Data":"51ff77af554af591986a3b4c271df334a6923528f713e8d3b7e658c37c7cfce7"}
Jul 17 12:46:50.573797 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:50.573644    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d" event={"ID":"1664b90e-7ff9-4901-ab0c-0df556f4f295","Type":"ContainerStarted","Data":"1b43fedd90863e2aa662262e78c9deac27e842eb9fd32d01d5655bb295bca056"}
Jul 17 12:46:50.592656 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:50.592598    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d" podStartSLOduration=1.388553129 podStartE2EDuration="1.592581972s" totalImagesPullingTime="204.028843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:46:50.590567558 +0000 UTC m=+1454.721952221" watchObservedRunningTime="2026-07-17 12:46:50.592581972 +0000 UTC m=+1454.723966632"
Jul 17 12:46:53.591073 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:53.591038    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:46:58.860887 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:58.860850    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:46:59.992828 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:59.992798    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:46:59.993221 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:46:59.992847    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:47:00.005400 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:00.005370    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:47:00.649134 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:00.649104    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-crf9d"
Jul 17 12:47:02.970597 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:02.970564    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:47:10.764144 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.764114    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"]
Jul 17 12:47:10.767650 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.767634    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:10.770436 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.770414    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 17 12:47:10.775282 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.775261    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"]
Jul 17 12:47:10.783455 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.783432    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:47:10.878590 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.878554    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2796cbe3-9897-47e0-95c6-12adf98aceab-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2\" (UID: \"2796cbe3-9897-47e0-95c6-12adf98aceab\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:10.878756 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.878603    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2796cbe3-9897-47e0-95c6-12adf98aceab-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2\" (UID: \"2796cbe3-9897-47e0-95c6-12adf98aceab\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:10.878756 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.878625    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2796cbe3-9897-47e0-95c6-12adf98aceab-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2\" (UID: \"2796cbe3-9897-47e0-95c6-12adf98aceab\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:10.878756 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.878670    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2796cbe3-9897-47e0-95c6-12adf98aceab-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2\" (UID: \"2796cbe3-9897-47e0-95c6-12adf98aceab\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:10.878756 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.878704    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2796cbe3-9897-47e0-95c6-12adf98aceab-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2\" (UID: \"2796cbe3-9897-47e0-95c6-12adf98aceab\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:10.878898 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.878786    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5tc9\" (UniqueName: \"kubernetes.io/projected/2796cbe3-9897-47e0-95c6-12adf98aceab-kube-api-access-x5tc9\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2\" (UID: \"2796cbe3-9897-47e0-95c6-12adf98aceab\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:10.980110 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.980076    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2796cbe3-9897-47e0-95c6-12adf98aceab-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2\" (UID: \"2796cbe3-9897-47e0-95c6-12adf98aceab\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:10.980278 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.980139    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5tc9\" (UniqueName: \"kubernetes.io/projected/2796cbe3-9897-47e0-95c6-12adf98aceab-kube-api-access-x5tc9\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2\" (UID: \"2796cbe3-9897-47e0-95c6-12adf98aceab\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:10.980278 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.980192    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2796cbe3-9897-47e0-95c6-12adf98aceab-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2\" (UID: \"2796cbe3-9897-47e0-95c6-12adf98aceab\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:10.980278 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.980251    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2796cbe3-9897-47e0-95c6-12adf98aceab-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2\" (UID: \"2796cbe3-9897-47e0-95c6-12adf98aceab\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:10.980460 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.980282    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2796cbe3-9897-47e0-95c6-12adf98aceab-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2\" (UID: \"2796cbe3-9897-47e0-95c6-12adf98aceab\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:10.980460 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.980302    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2796cbe3-9897-47e0-95c6-12adf98aceab-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2\" (UID: \"2796cbe3-9897-47e0-95c6-12adf98aceab\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:10.980542 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.980505    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2796cbe3-9897-47e0-95c6-12adf98aceab-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2\" (UID: \"2796cbe3-9897-47e0-95c6-12adf98aceab\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:10.980577 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.980557    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2796cbe3-9897-47e0-95c6-12adf98aceab-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2\" (UID: \"2796cbe3-9897-47e0-95c6-12adf98aceab\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:10.980700 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.980679    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2796cbe3-9897-47e0-95c6-12adf98aceab-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2\" (UID: \"2796cbe3-9897-47e0-95c6-12adf98aceab\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:10.982553 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.982527    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2796cbe3-9897-47e0-95c6-12adf98aceab-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2\" (UID: \"2796cbe3-9897-47e0-95c6-12adf98aceab\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:10.982948 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.982927    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2796cbe3-9897-47e0-95c6-12adf98aceab-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2\" (UID: \"2796cbe3-9897-47e0-95c6-12adf98aceab\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:10.988698 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:10.988678    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5tc9\" (UniqueName: \"kubernetes.io/projected/2796cbe3-9897-47e0-95c6-12adf98aceab-kube-api-access-x5tc9\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2\" (UID: \"2796cbe3-9897-47e0-95c6-12adf98aceab\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:11.081456 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:11.081430    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:11.217769 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:11.217741    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"]
Jul 17 12:47:11.218880 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:47:11.218840    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2796cbe3_9897_47e0_95c6_12adf98aceab.slice/crio-21cc67912251597e3cbf8afaa6c17482fce7f5fcc855f0f0cf48e467439c3cb5 WatchSource:0}: Error finding container 21cc67912251597e3cbf8afaa6c17482fce7f5fcc855f0f0cf48e467439c3cb5: Status 404 returned error can't find the container with id 21cc67912251597e3cbf8afaa6c17482fce7f5fcc855f0f0cf48e467439c3cb5
Jul 17 12:47:11.708353 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:11.708323    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2" event={"ID":"2796cbe3-9897-47e0-95c6-12adf98aceab","Type":"ContainerStarted","Data":"be408572b66df468ad10a8fa5113b3ec7f78b498e9fcc18323d58b7b00b86541"}
Jul 17 12:47:11.708540 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:11.708364    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2" event={"ID":"2796cbe3-9897-47e0-95c6-12adf98aceab","Type":"ContainerStarted","Data":"21cc67912251597e3cbf8afaa6c17482fce7f5fcc855f0f0cf48e467439c3cb5"}
Jul 17 12:47:11.726835 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:11.726734    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2" podStartSLOduration=1.482617872 podStartE2EDuration="1.726712218s" totalImagesPullingTime="244.094346ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:47:11.726170587 +0000 UTC m=+1475.857555268" watchObservedRunningTime="2026-07-17 12:47:11.726712218 +0000 UTC m=+1475.858096880"
Jul 17 12:47:21.081950 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:21.081899    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:21.082504 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:21.082198    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:21.094595 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:21.094567    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:21.778527 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:21.778503    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2"
Jul 17 12:47:36.411020 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:36.410972    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 12:47:36.414990 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:36.414963    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 12:47:36.416368 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:36.416349    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 12:47:36.419566 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:36.419545    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 12:47:50.687334 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:50.687304    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:47:50.740232 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:47:50.740207    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:48:33.300377 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:48:33.300340    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:48:34.930464 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:48:34.930435    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:49:20.910311 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:49:20.910278    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-85456b7f49-84lhh"]
Jul 17 12:49:20.910954 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:49:20.910573    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-85456b7f49-84lhh" podUID="fc8d3096-a73a-4d3b-8836-745f11003ab7" containerName="manager" containerID="cri-o://b3e97e498fd09e84722c4aa45d7773083c34b772c97fef95a2c1be086990ab5b" gracePeriod=30
Jul 17 12:49:21.149227 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:49:21.149206    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85456b7f49-84lhh"
Jul 17 12:49:21.257768 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:49:21.257699    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc8d3096-a73a-4d3b-8836-745f11003ab7-webhook-cert\") pod \"fc8d3096-a73a-4d3b-8836-745f11003ab7\" (UID: \"fc8d3096-a73a-4d3b-8836-745f11003ab7\") "
Jul 17 12:49:21.257768 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:49:21.257733    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq9ph\" (UniqueName: \"kubernetes.io/projected/fc8d3096-a73a-4d3b-8836-745f11003ab7-kube-api-access-dq9ph\") pod \"fc8d3096-a73a-4d3b-8836-745f11003ab7\" (UID: \"fc8d3096-a73a-4d3b-8836-745f11003ab7\") "
Jul 17 12:49:21.259747 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:49:21.259721    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc8d3096-a73a-4d3b-8836-745f11003ab7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fc8d3096-a73a-4d3b-8836-745f11003ab7" (UID: "fc8d3096-a73a-4d3b-8836-745f11003ab7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:49:21.259859 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:49:21.259751    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8d3096-a73a-4d3b-8836-745f11003ab7-kube-api-access-dq9ph" (OuterVolumeSpecName: "kube-api-access-dq9ph") pod "fc8d3096-a73a-4d3b-8836-745f11003ab7" (UID: "fc8d3096-a73a-4d3b-8836-745f11003ab7"). InnerVolumeSpecName "kube-api-access-dq9ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:49:21.359246 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:49:21.359225    2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc8d3096-a73a-4d3b-8836-745f11003ab7-webhook-cert\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:49:21.359246 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:49:21.359245    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq9ph\" (UniqueName: \"kubernetes.io/projected/fc8d3096-a73a-4d3b-8836-745f11003ab7-kube-api-access-dq9ph\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:49:21.445752 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:49:21.445727    2565 generic.go:356] "Generic (PLEG): container finished" podID="fc8d3096-a73a-4d3b-8836-745f11003ab7" containerID="b3e97e498fd09e84722c4aa45d7773083c34b772c97fef95a2c1be086990ab5b" exitCode=0
Jul 17 12:49:21.445846 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:49:21.445772    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85456b7f49-84lhh"
Jul 17 12:49:21.445846 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:49:21.445815    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85456b7f49-84lhh" event={"ID":"fc8d3096-a73a-4d3b-8836-745f11003ab7","Type":"ContainerDied","Data":"b3e97e498fd09e84722c4aa45d7773083c34b772c97fef95a2c1be086990ab5b"}
Jul 17 12:49:21.445918 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:49:21.445858    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85456b7f49-84lhh" event={"ID":"fc8d3096-a73a-4d3b-8836-745f11003ab7","Type":"ContainerDied","Data":"c02579166843f0449acf2b4ceae5867f393f1483665bb06c16e2d89f1f40ead7"}
Jul 17 12:49:21.445918 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:49:21.445880    2565 scope.go:117] "RemoveContainer" containerID="b3e97e498fd09e84722c4aa45d7773083c34b772c97fef95a2c1be086990ab5b"
Jul 17 12:49:21.456030 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:49:21.455971    2565 scope.go:117] "RemoveContainer" containerID="b3e97e498fd09e84722c4aa45d7773083c34b772c97fef95a2c1be086990ab5b"
Jul 17 12:49:21.456300 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:49:21.456282    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3e97e498fd09e84722c4aa45d7773083c34b772c97fef95a2c1be086990ab5b\": container with ID starting with b3e97e498fd09e84722c4aa45d7773083c34b772c97fef95a2c1be086990ab5b not found: ID does not exist" containerID="b3e97e498fd09e84722c4aa45d7773083c34b772c97fef95a2c1be086990ab5b"
Jul 17 12:49:21.456350 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:49:21.456308    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e97e498fd09e84722c4aa45d7773083c34b772c97fef95a2c1be086990ab5b"} err="failed to get container status \"b3e97e498fd09e84722c4aa45d7773083c34b772c97fef95a2c1be086990ab5b\": rpc error: code = NotFound desc = could not find container \"b3e97e498fd09e84722c4aa45d7773083c34b772c97fef95a2c1be086990ab5b\": container with ID starting with b3e97e498fd09e84722c4aa45d7773083c34b772c97fef95a2c1be086990ab5b not found: ID does not exist"
Jul 17 12:49:21.507748 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:49:21.507703    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-85456b7f49-84lhh"]
Jul 17 12:49:21.510345 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:49:21.510297    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-85456b7f49-84lhh"]
Jul 17 12:49:22.403241 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:49:22.403203    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc8d3096-a73a-4d3b-8836-745f11003ab7" path="/var/lib/kubelet/pods/fc8d3096-a73a-4d3b-8836-745f11003ab7/volumes"
Jul 17 12:49:55.204377 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:49:55.204343    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:50:22.393772 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:22.393740    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-85456b7f49-f7b5c"]
Jul 17 12:50:22.394178 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:22.394165    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc8d3096-a73a-4d3b-8836-745f11003ab7" containerName="manager"
Jul 17 12:50:22.394178 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:22.394179    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc8d3096-a73a-4d3b-8836-745f11003ab7" containerName="manager"
Jul 17 12:50:22.394249 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:22.394240    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc8d3096-a73a-4d3b-8836-745f11003ab7" containerName="manager"
Jul 17 12:50:22.397363 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:22.397345    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85456b7f49-f7b5c"
Jul 17 12:50:22.400125 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:22.400105    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 17 12:50:22.401185 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:22.401167    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 17 12:50:22.401280 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:22.401174    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2bz89\""
Jul 17 12:50:22.407567 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:22.406476    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85456b7f49-f7b5c"]
Jul 17 12:50:22.457531 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:22.457503    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df92282c-d4f9-40d7-a27b-897e6d9def36-webhook-cert\") pod \"maas-controller-85456b7f49-f7b5c\" (UID: \"df92282c-d4f9-40d7-a27b-897e6d9def36\") " pod="opendatahub/maas-controller-85456b7f49-f7b5c"
Jul 17 12:50:22.457531 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:22.457532    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfhcz\" (UniqueName: \"kubernetes.io/projected/df92282c-d4f9-40d7-a27b-897e6d9def36-kube-api-access-qfhcz\") pod \"maas-controller-85456b7f49-f7b5c\" (UID: \"df92282c-d4f9-40d7-a27b-897e6d9def36\") " pod="opendatahub/maas-controller-85456b7f49-f7b5c"
Jul 17 12:50:22.558372 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:22.558347    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df92282c-d4f9-40d7-a27b-897e6d9def36-webhook-cert\") pod \"maas-controller-85456b7f49-f7b5c\" (UID: \"df92282c-d4f9-40d7-a27b-897e6d9def36\") " pod="opendatahub/maas-controller-85456b7f49-f7b5c"
Jul 17 12:50:22.558499 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:22.558380    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfhcz\" (UniqueName: \"kubernetes.io/projected/df92282c-d4f9-40d7-a27b-897e6d9def36-kube-api-access-qfhcz\") pod \"maas-controller-85456b7f49-f7b5c\" (UID: \"df92282c-d4f9-40d7-a27b-897e6d9def36\") " pod="opendatahub/maas-controller-85456b7f49-f7b5c"
Jul 17 12:50:22.560748 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:22.560728    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df92282c-d4f9-40d7-a27b-897e6d9def36-webhook-cert\") pod \"maas-controller-85456b7f49-f7b5c\" (UID: \"df92282c-d4f9-40d7-a27b-897e6d9def36\") " pod="opendatahub/maas-controller-85456b7f49-f7b5c"
Jul 17 12:50:22.566403 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:22.566383    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfhcz\" (UniqueName: \"kubernetes.io/projected/df92282c-d4f9-40d7-a27b-897e6d9def36-kube-api-access-qfhcz\") pod \"maas-controller-85456b7f49-f7b5c\" (UID: \"df92282c-d4f9-40d7-a27b-897e6d9def36\") " pod="opendatahub/maas-controller-85456b7f49-f7b5c"
Jul 17 12:50:22.709584 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:22.709516    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85456b7f49-f7b5c"
Jul 17 12:50:22.842639 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:22.842613    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85456b7f49-f7b5c"]
Jul 17 12:50:22.843952 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:50:22.843930    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf92282c_d4f9_40d7_a27b_897e6d9def36.slice/crio-03d79201d102312efdcb7ebf057ee70697af6b2bdf1688864ca4d9d8f63b70c1 WatchSource:0}: Error finding container 03d79201d102312efdcb7ebf057ee70697af6b2bdf1688864ca4d9d8f63b70c1: Status 404 returned error can't find the container with id 03d79201d102312efdcb7ebf057ee70697af6b2bdf1688864ca4d9d8f63b70c1
Jul 17 12:50:22.845282 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:22.845263    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 12:50:23.796862 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:23.796825    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85456b7f49-f7b5c" event={"ID":"df92282c-d4f9-40d7-a27b-897e6d9def36","Type":"ContainerStarted","Data":"461f150b95525b96f025da13cd1012a19a2c7a1c02193031ca67a6d034e4149f"}
Jul 17 12:50:23.797226 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:23.796873    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85456b7f49-f7b5c" event={"ID":"df92282c-d4f9-40d7-a27b-897e6d9def36","Type":"ContainerStarted","Data":"03d79201d102312efdcb7ebf057ee70697af6b2bdf1688864ca4d9d8f63b70c1"}
Jul 17 12:50:23.797226 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:23.796966    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-85456b7f49-f7b5c"
Jul 17 12:50:23.816822 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:23.816740    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-85456b7f49-f7b5c" podStartSLOduration=1.112328717 podStartE2EDuration="1.816726745s" totalImagesPullingTime="704.398028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:50:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:50:23.815726237 +0000 UTC m=+1667.947110896" watchObservedRunningTime="2026-07-17 12:50:23.816726745 +0000 UTC m=+1667.948111403"
Jul 17 12:50:34.805786 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:34.805756    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-85456b7f49-f7b5c"
Jul 17 12:50:39.378628 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:39.378588    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:50:39.503590 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:39.503560    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-85456b7f49-f7b5c"]
Jul 17 12:50:39.503745 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:39.503726    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-85456b7f49-f7b5c" podUID="df92282c-d4f9-40d7-a27b-897e6d9def36" containerName="manager" containerID="cri-o://461f150b95525b96f025da13cd1012a19a2c7a1c02193031ca67a6d034e4149f" gracePeriod=30
Jul 17 12:50:39.755920 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:39.755899    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85456b7f49-f7b5c"
Jul 17 12:50:39.889938 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:39.889906    2565 generic.go:356] "Generic (PLEG): container finished" podID="df92282c-d4f9-40d7-a27b-897e6d9def36" containerID="461f150b95525b96f025da13cd1012a19a2c7a1c02193031ca67a6d034e4149f" exitCode=0
Jul 17 12:50:39.890111 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:39.889946    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85456b7f49-f7b5c"
Jul 17 12:50:39.890111 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:39.889961    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85456b7f49-f7b5c" event={"ID":"df92282c-d4f9-40d7-a27b-897e6d9def36","Type":"ContainerDied","Data":"461f150b95525b96f025da13cd1012a19a2c7a1c02193031ca67a6d034e4149f"}
Jul 17 12:50:39.890111 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:39.890019    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85456b7f49-f7b5c" event={"ID":"df92282c-d4f9-40d7-a27b-897e6d9def36","Type":"ContainerDied","Data":"03d79201d102312efdcb7ebf057ee70697af6b2bdf1688864ca4d9d8f63b70c1"}
Jul 17 12:50:39.890111 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:39.890041    2565 scope.go:117] "RemoveContainer" containerID="461f150b95525b96f025da13cd1012a19a2c7a1c02193031ca67a6d034e4149f"
Jul 17 12:50:39.899210 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:39.899195    2565 scope.go:117] "RemoveContainer" containerID="461f150b95525b96f025da13cd1012a19a2c7a1c02193031ca67a6d034e4149f"
Jul 17 12:50:39.899458 ip-10-0-134-119 kubenswrapper[2565]: E0717 12:50:39.899438    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"461f150b95525b96f025da13cd1012a19a2c7a1c02193031ca67a6d034e4149f\": container with ID starting with 461f150b95525b96f025da13cd1012a19a2c7a1c02193031ca67a6d034e4149f not found: ID does not exist" containerID="461f150b95525b96f025da13cd1012a19a2c7a1c02193031ca67a6d034e4149f"
Jul 17 12:50:39.899528 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:39.899470    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461f150b95525b96f025da13cd1012a19a2c7a1c02193031ca67a6d034e4149f"} err="failed to get container status \"461f150b95525b96f025da13cd1012a19a2c7a1c02193031ca67a6d034e4149f\": rpc error: code = NotFound desc = could not find container \"461f150b95525b96f025da13cd1012a19a2c7a1c02193031ca67a6d034e4149f\": container with ID starting with 461f150b95525b96f025da13cd1012a19a2c7a1c02193031ca67a6d034e4149f not found: ID does not exist"
Jul 17 12:50:39.900778 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:39.900759    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfhcz\" (UniqueName: \"kubernetes.io/projected/df92282c-d4f9-40d7-a27b-897e6d9def36-kube-api-access-qfhcz\") pod \"df92282c-d4f9-40d7-a27b-897e6d9def36\" (UID: \"df92282c-d4f9-40d7-a27b-897e6d9def36\") "
Jul 17 12:50:39.900854 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:39.900822    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df92282c-d4f9-40d7-a27b-897e6d9def36-webhook-cert\") pod \"df92282c-d4f9-40d7-a27b-897e6d9def36\" (UID: \"df92282c-d4f9-40d7-a27b-897e6d9def36\") "
Jul 17 12:50:39.902732 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:39.902707    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df92282c-d4f9-40d7-a27b-897e6d9def36-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "df92282c-d4f9-40d7-a27b-897e6d9def36" (UID: "df92282c-d4f9-40d7-a27b-897e6d9def36"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 12:50:39.902888 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:39.902869    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df92282c-d4f9-40d7-a27b-897e6d9def36-kube-api-access-qfhcz" (OuterVolumeSpecName: "kube-api-access-qfhcz") pod "df92282c-d4f9-40d7-a27b-897e6d9def36" (UID: "df92282c-d4f9-40d7-a27b-897e6d9def36"). InnerVolumeSpecName "kube-api-access-qfhcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:50:40.002187 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:40.002123    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfhcz\" (UniqueName: \"kubernetes.io/projected/df92282c-d4f9-40d7-a27b-897e6d9def36-kube-api-access-qfhcz\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:50:40.002187 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:40.002147    2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df92282c-d4f9-40d7-a27b-897e6d9def36-webhook-cert\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:50:40.214037 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:40.213895    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-85456b7f49-f7b5c"]
Jul 17 12:50:40.216827 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:40.216804    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-85456b7f49-f7b5c"]
Jul 17 12:50:40.402578 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:40.402549    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df92282c-d4f9-40d7-a27b-897e6d9def36" path="/var/lib/kubelet/pods/df92282c-d4f9-40d7-a27b-897e6d9def36/volumes"
Jul 17 12:50:40.939706 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:40.939671    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:50:41.061950 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:41.061922    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-85456b7f49-6mkfq"]
Jul 17 12:50:41.062348 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:41.062336    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df92282c-d4f9-40d7-a27b-897e6d9def36" containerName="manager"
Jul 17 12:50:41.062391 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:41.062350    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="df92282c-d4f9-40d7-a27b-897e6d9def36" containerName="manager"
Jul 17 12:50:41.062452 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:41.062444    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="df92282c-d4f9-40d7-a27b-897e6d9def36" containerName="manager"
Jul 17 12:50:41.066924 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:41.066907    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85456b7f49-6mkfq"
Jul 17 12:50:41.069838 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:41.069818    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 17 12:50:41.071025 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:41.071008    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 17 12:50:41.071115 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:41.071045    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2bz89\""
Jul 17 12:50:41.076617 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:41.076599    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85456b7f49-6mkfq"]
Jul 17 12:50:41.215298 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:41.215201    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7c0a4cb-c50f-488f-9758-bfac25466cd7-webhook-cert\") pod \"maas-controller-85456b7f49-6mkfq\" (UID: \"d7c0a4cb-c50f-488f-9758-bfac25466cd7\") " pod="opendatahub/maas-controller-85456b7f49-6mkfq"
Jul 17 12:50:41.215298 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:41.215277    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcljd\" (UniqueName: \"kubernetes.io/projected/d7c0a4cb-c50f-488f-9758-bfac25466cd7-kube-api-access-fcljd\") pod \"maas-controller-85456b7f49-6mkfq\" (UID: \"d7c0a4cb-c50f-488f-9758-bfac25466cd7\") " pod="opendatahub/maas-controller-85456b7f49-6mkfq"
Jul 17 12:50:41.316431 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:41.316397    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcljd\" (UniqueName: \"kubernetes.io/projected/d7c0a4cb-c50f-488f-9758-bfac25466cd7-kube-api-access-fcljd\") pod \"maas-controller-85456b7f49-6mkfq\" (UID: \"d7c0a4cb-c50f-488f-9758-bfac25466cd7\") " pod="opendatahub/maas-controller-85456b7f49-6mkfq"
Jul 17 12:50:41.316619 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:41.316454    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7c0a4cb-c50f-488f-9758-bfac25466cd7-webhook-cert\") pod \"maas-controller-85456b7f49-6mkfq\" (UID: \"d7c0a4cb-c50f-488f-9758-bfac25466cd7\") " pod="opendatahub/maas-controller-85456b7f49-6mkfq"
Jul 17 12:50:41.318912 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:41.318887    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7c0a4cb-c50f-488f-9758-bfac25466cd7-webhook-cert\") pod \"maas-controller-85456b7f49-6mkfq\" (UID: \"d7c0a4cb-c50f-488f-9758-bfac25466cd7\") " pod="opendatahub/maas-controller-85456b7f49-6mkfq"
Jul 17 12:50:41.324696 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:41.324678    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcljd\" (UniqueName: \"kubernetes.io/projected/d7c0a4cb-c50f-488f-9758-bfac25466cd7-kube-api-access-fcljd\") pod \"maas-controller-85456b7f49-6mkfq\" (UID: \"d7c0a4cb-c50f-488f-9758-bfac25466cd7\") " pod="opendatahub/maas-controller-85456b7f49-6mkfq"
Jul 17 12:50:41.378740 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:41.378718    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85456b7f49-6mkfq"
Jul 17 12:50:41.520337 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:41.520311    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85456b7f49-6mkfq"]
Jul 17 12:50:41.520726 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:50:41.520412    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c0a4cb_c50f_488f_9758_bfac25466cd7.slice/crio-383ef48f6c89d231b74c2184b1721f8621e331ebd7ef89cb4fe333ce1192d312 WatchSource:0}: Error finding container 383ef48f6c89d231b74c2184b1721f8621e331ebd7ef89cb4fe333ce1192d312: Status 404 returned error can't find the container with id 383ef48f6c89d231b74c2184b1721f8621e331ebd7ef89cb4fe333ce1192d312
Jul 17 12:50:41.904617 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:41.904581    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85456b7f49-6mkfq" event={"ID":"d7c0a4cb-c50f-488f-9758-bfac25466cd7","Type":"ContainerStarted","Data":"383ef48f6c89d231b74c2184b1721f8621e331ebd7ef89cb4fe333ce1192d312"}
Jul 17 12:50:42.911592 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:42.911559    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85456b7f49-6mkfq" event={"ID":"d7c0a4cb-c50f-488f-9758-bfac25466cd7","Type":"ContainerStarted","Data":"040935e613b08f1488ebeaedfddb14a18f9a47c7f2ca2900ef0ceb06e630320c"}
Jul 17 12:50:42.912002 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:42.911746    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-85456b7f49-6mkfq"
Jul 17 12:50:42.934465 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:42.934418    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-85456b7f49-6mkfq" podStartSLOduration=1.534949574 podStartE2EDuration="1.934405322s" totalImagesPullingTime="399.455748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:50:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:50:42.933115357 +0000 UTC m=+1687.064500016" watchObservedRunningTime="2026-07-17 12:50:42.934405322 +0000 UTC m=+1687.065789981"
Jul 17 12:50:53.919888 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:50:53.919856    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-85456b7f49-6mkfq"
Jul 17 12:51:17.383680 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:51:17.383608    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:51:17.472784 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:51:17.472759    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:51:17.702349 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:51:17.702255    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:51:24.127797 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:51:24.127761    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:51:24.262290 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:51:24.262250    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:51:24.402952 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:51:24.402866    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:51:24.611421 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:51:24.611387    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:51:30.272369 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:51:30.272332    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:51:30.334847 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:51:30.334817    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:52:23.459464 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:52:23.459430    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:52:23.560353 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:52:23.560319    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:52:36.450428 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:52:36.450402    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 12:52:36.456211 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:52:36.456189    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 12:52:36.456411 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:52:36.456391    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 12:52:36.461256 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:52:36.461241    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 12:54:36.007305 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:54:36.007272    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:54:37.848224 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:54:37.848190    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:54:42.564971 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:54:42.564938    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:54:42.621278 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:54:42.621244    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:55:29.711640 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:55:29.711559    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl"]
Jul 17 12:55:29.712396 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:55:29.711753    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl" podUID="50f68e7a-3d6a-4da2-9264-06fec2b550e6" containerName="manager" containerID="cri-o://153344f5522650cef149454e41d0fd12043ca807fc091d52f01b19acc48fcd3f" gracePeriod=10
Jul 17 12:55:32.564566 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:55:32.564540    2565 generic.go:356] "Generic (PLEG): container finished" podID="50f68e7a-3d6a-4da2-9264-06fec2b550e6" containerID="153344f5522650cef149454e41d0fd12043ca807fc091d52f01b19acc48fcd3f" exitCode=0
Jul 17 12:55:32.564940 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:55:32.564608    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl" event={"ID":"50f68e7a-3d6a-4da2-9264-06fec2b550e6","Type":"ContainerDied","Data":"153344f5522650cef149454e41d0fd12043ca807fc091d52f01b19acc48fcd3f"}
Jul 17 12:55:32.564940 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:55:32.564662    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl" event={"ID":"50f68e7a-3d6a-4da2-9264-06fec2b550e6","Type":"ContainerDied","Data":"ddcbf45b03a6d2281caa28c07de8c1ddbcea13d3148d999db68306d95eecec61"}
Jul 17 12:55:32.564940 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:55:32.564679    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddcbf45b03a6d2281caa28c07de8c1ddbcea13d3148d999db68306d95eecec61"
Jul 17 12:55:32.573878 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:55:32.573861    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl"
Jul 17 12:55:32.748972 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:55:32.748898    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47fd6\" (UniqueName: \"kubernetes.io/projected/50f68e7a-3d6a-4da2-9264-06fec2b550e6-kube-api-access-47fd6\") pod \"50f68e7a-3d6a-4da2-9264-06fec2b550e6\" (UID: \"50f68e7a-3d6a-4da2-9264-06fec2b550e6\") "
Jul 17 12:55:32.749129 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:55:32.748972    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50f68e7a-3d6a-4da2-9264-06fec2b550e6-extensions-socket-volume\") pod \"50f68e7a-3d6a-4da2-9264-06fec2b550e6\" (UID: \"50f68e7a-3d6a-4da2-9264-06fec2b550e6\") "
Jul 17 12:55:32.749318 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:55:32.749296    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f68e7a-3d6a-4da2-9264-06fec2b550e6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "50f68e7a-3d6a-4da2-9264-06fec2b550e6" (UID: "50f68e7a-3d6a-4da2-9264-06fec2b550e6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 12:55:32.751061 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:55:32.751039    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f68e7a-3d6a-4da2-9264-06fec2b550e6-kube-api-access-47fd6" (OuterVolumeSpecName: "kube-api-access-47fd6") pod "50f68e7a-3d6a-4da2-9264-06fec2b550e6" (UID: "50f68e7a-3d6a-4da2-9264-06fec2b550e6"). InnerVolumeSpecName "kube-api-access-47fd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 12:55:32.850086 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:55:32.850064    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50f68e7a-3d6a-4da2-9264-06fec2b550e6-extensions-socket-volume\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:55:32.850086 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:55:32.850083    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47fd6\" (UniqueName: \"kubernetes.io/projected/50f68e7a-3d6a-4da2-9264-06fec2b550e6-kube-api-access-47fd6\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 12:55:33.548709 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:55:33.548670    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl" podUID="50f68e7a-3d6a-4da2-9264-06fec2b550e6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.46:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 17 12:55:33.570828 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:55:33.570804    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl"
Jul 17 12:55:33.593281 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:55:33.593261    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl"]
Jul 17 12:55:33.596794 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:55:33.596768    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hlkvl"]
Jul 17 12:55:34.403758 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:55:34.403730    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50f68e7a-3d6a-4da2-9264-06fec2b550e6" path="/var/lib/kubelet/pods/50f68e7a-3d6a-4da2-9264-06fec2b550e6/volumes"
Jul 17 12:55:36.658410 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:55:36.658378    2565 scope.go:117] "RemoveContainer" containerID="153344f5522650cef149454e41d0fd12043ca807fc091d52f01b19acc48fcd3f"
Jul 17 12:56:11.161161 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:11.161125    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:56:30.963263 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:30.963228    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-766lt"]
Jul 17 12:56:30.963663 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:30.963648    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f68e7a-3d6a-4da2-9264-06fec2b550e6" containerName="manager"
Jul 17 12:56:30.963663 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:30.963663    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f68e7a-3d6a-4da2-9264-06fec2b550e6" containerName="manager"
Jul 17 12:56:30.963775 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:30.963729    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="50f68e7a-3d6a-4da2-9264-06fec2b550e6" containerName="manager"
Jul 17 12:56:30.966966 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:30.966949    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-766lt"
Jul 17 12:56:30.970088 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:30.970066    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qv7t7\""
Jul 17 12:56:30.978068 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:30.978045    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-766lt"]
Jul 17 12:56:31.128350 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:31.128323    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtz85\" (UniqueName: \"kubernetes.io/projected/45c88802-f8b6-4983-a40d-39523cc5c217-kube-api-access-xtz85\") pod \"kuadrant-operator-controller-manager-86878c769f-766lt\" (UID: \"45c88802-f8b6-4983-a40d-39523cc5c217\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-766lt"
Jul 17 12:56:31.128492 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:31.128354    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45c88802-f8b6-4983-a40d-39523cc5c217-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-766lt\" (UID: \"45c88802-f8b6-4983-a40d-39523cc5c217\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-766lt"
Jul 17 12:56:31.229725 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:31.229633    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtz85\" (UniqueName: \"kubernetes.io/projected/45c88802-f8b6-4983-a40d-39523cc5c217-kube-api-access-xtz85\") pod \"kuadrant-operator-controller-manager-86878c769f-766lt\" (UID: \"45c88802-f8b6-4983-a40d-39523cc5c217\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-766lt"
Jul 17 12:56:31.229725 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:31.229674    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45c88802-f8b6-4983-a40d-39523cc5c217-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-766lt\" (UID: \"45c88802-f8b6-4983-a40d-39523cc5c217\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-766lt"
Jul 17 12:56:31.230023 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:31.230007    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45c88802-f8b6-4983-a40d-39523cc5c217-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-766lt\" (UID: \"45c88802-f8b6-4983-a40d-39523cc5c217\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-766lt"
Jul 17 12:56:31.238223 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:31.238200    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtz85\" (UniqueName: \"kubernetes.io/projected/45c88802-f8b6-4983-a40d-39523cc5c217-kube-api-access-xtz85\") pod \"kuadrant-operator-controller-manager-86878c769f-766lt\" (UID: \"45c88802-f8b6-4983-a40d-39523cc5c217\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-766lt"
Jul 17 12:56:31.277929 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:31.277903    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-766lt"
Jul 17 12:56:31.423687 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:31.423660    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-766lt"]
Jul 17 12:56:31.425156 ip-10-0-134-119 kubenswrapper[2565]: W0717 12:56:31.425129    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c88802_f8b6_4983_a40d_39523cc5c217.slice/crio-21605ef04cb2c08fd14165c57306d31467d4defce9505abc86c11983105ee517 WatchSource:0}: Error finding container 21605ef04cb2c08fd14165c57306d31467d4defce9505abc86c11983105ee517: Status 404 returned error can't find the container with id 21605ef04cb2c08fd14165c57306d31467d4defce9505abc86c11983105ee517
Jul 17 12:56:31.911136 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:31.911100    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-766lt" event={"ID":"45c88802-f8b6-4983-a40d-39523cc5c217","Type":"ContainerStarted","Data":"ebb8bd3953aafe27ec7abce494591e60049b6a59dcf89070adce571e4a3f29b4"}
Jul 17 12:56:31.911328 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:31.911149    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-766lt" event={"ID":"45c88802-f8b6-4983-a40d-39523cc5c217","Type":"ContainerStarted","Data":"21605ef04cb2c08fd14165c57306d31467d4defce9505abc86c11983105ee517"}
Jul 17 12:56:31.911328 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:31.911207    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-766lt"
Jul 17 12:56:31.935387 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:31.935344    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-766lt" podStartSLOduration=1.9353305010000001 podStartE2EDuration="1.935330501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 12:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 12:56:31.933902153 +0000 UTC m=+2036.065286813" watchObservedRunningTime="2026-07-17 12:56:31.935330501 +0000 UTC m=+2036.066715163"
Jul 17 12:56:40.098749 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:40.098714    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:56:42.919527 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:42.919499    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-766lt"
Jul 17 12:56:44.954712 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:44.954678    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:56:45.034737 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:56:45.034703    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 12:57:36.489528 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:57:36.489503    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 12:57:36.494534 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:57:36.494515    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 12:57:36.498430 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:57:36.498413    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 12:57:36.506735 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:57:36.506712    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 12:59:56.110364 ip-10-0-134-119 kubenswrapper[2565]: I0717 12:59:56.110328    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:00:00.146541 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:00:00.146469    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738220-ffqq8"]
Jul 17 13:00:00.150495 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:00:00.150474    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738220-ffqq8"
Jul 17 13:00:00.153194 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:00:00.153171    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:00:00.160272 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:00:00.160252    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738220-ffqq8"
Jul 17 13:00:00.187158 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:00:00.187123    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738220-ffqq8"]
Jul 17 13:00:00.305710 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:00:00.305680    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738220-ffqq8"]
Jul 17 13:00:00.306717 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:00:00.306692    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fed3c4c_ed5c_4976_ad2a_969a1a3c3d13.slice/crio-d3b63d2b0b86ecc3e79e893dcb292716832c6ab88c047815fe0389789f73481e WatchSource:0}: Error finding container d3b63d2b0b86ecc3e79e893dcb292716832c6ab88c047815fe0389789f73481e: Status 404 returned error can't find the container with id d3b63d2b0b86ecc3e79e893dcb292716832c6ab88c047815fe0389789f73481e
Jul 17 13:00:00.308715 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:00:00.308699    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 13:00:00.379491 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:00:00.379446    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:00:01.157279 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:00:01.157241    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738220-ffqq8" event={"ID":"9fed3c4c-ed5c-4976-ad2a-969a1a3c3d13","Type":"ContainerStarted","Data":"d3b63d2b0b86ecc3e79e893dcb292716832c6ab88c047815fe0389789f73481e"}
Jul 17 13:00:04.178974 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:00:04.178935    2565 generic.go:356] "Generic (PLEG): container finished" podID="9fed3c4c-ed5c-4976-ad2a-969a1a3c3d13" containerID="bbabf9a138a48e45633ec893df0ef43175cd2befd40bb49b3b46a4b781e25d47" exitCode=0
Jul 17 13:00:04.179381 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:00:04.179025    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738220-ffqq8" event={"ID":"9fed3c4c-ed5c-4976-ad2a-969a1a3c3d13","Type":"ContainerDied","Data":"bbabf9a138a48e45633ec893df0ef43175cd2befd40bb49b3b46a4b781e25d47"}
Jul 17 13:00:05.333692 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:00:05.333667    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738220-ffqq8"
Jul 17 13:00:06.192241 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:00:06.192209    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738220-ffqq8" event={"ID":"9fed3c4c-ed5c-4976-ad2a-969a1a3c3d13","Type":"ContainerDied","Data":"d3b63d2b0b86ecc3e79e893dcb292716832c6ab88c047815fe0389789f73481e"}
Jul 17 13:00:06.192447 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:00:06.192257    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3b63d2b0b86ecc3e79e893dcb292716832c6ab88c047815fe0389789f73481e"
Jul 17 13:00:06.192447 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:00:06.192222    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738220-ffqq8"
Jul 17 13:01:56.712469 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:01:56.712426    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:02:28.321914 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:28.321871    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:02:28.398870 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:28.398837    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:02:28.774780 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:28.774691    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:02:34.827785 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:34.827744    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:02:34.928519 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:34.928492    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:02:35.055564 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:35.055530    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:02:35.155631 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:35.155558    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:02:36.258362 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.258325    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:02:36.347536 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.347503    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:02:36.527946 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.527868    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"]
Jul 17 13:02:36.529015 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.528964    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fed3c4c-ed5c-4976-ad2a-969a1a3c3d13" containerName="cleanup"
Jul 17 13:02:36.529194 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.529172    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fed3c4c-ed5c-4976-ad2a-969a1a3c3d13" containerName="cleanup"
Jul 17 13:02:36.529351 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.529334    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fed3c4c-ed5c-4976-ad2a-969a1a3c3d13" containerName="cleanup"
Jul 17 13:02:36.532891 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.532869    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:02:36.536089 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.535975    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3fc8989b-serving-cert\""
Jul 17 13:02:36.553872 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.553841    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"]
Jul 17 13:02:36.555712 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.555690    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 13:02:36.556646 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.556619    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkhnq\" (UniqueName: \"kubernetes.io/projected/c5b21a6e-2f4a-43ea-b85d-980b2169cea1-kube-api-access-xkhnq\") pod \"maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb\" (UID: \"c5b21a6e-2f4a-43ea-b85d-980b2169cea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:02:36.556751 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.556676    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5b21a6e-2f4a-43ea-b85d-980b2169cea1-maas-api-tls\") pod \"maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb\" (UID: \"c5b21a6e-2f4a-43ea-b85d-980b2169cea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:02:36.556751 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.556742    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5b21a6e-2f4a-43ea-b85d-980b2169cea1-openshift-service-ca\") pod \"maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb\" (UID: \"c5b21a6e-2f4a-43ea-b85d-980b2169cea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:02:36.561788 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.561761    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 13:02:36.566668 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.566647    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 13:02:36.571800 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.571784    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 13:02:36.574851 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.574828    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:02:36.657701 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.657676    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5b21a6e-2f4a-43ea-b85d-980b2169cea1-openshift-service-ca\") pod \"maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb\" (UID: \"c5b21a6e-2f4a-43ea-b85d-980b2169cea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:02:36.657814 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.657736    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkhnq\" (UniqueName: \"kubernetes.io/projected/c5b21a6e-2f4a-43ea-b85d-980b2169cea1-kube-api-access-xkhnq\") pod \"maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb\" (UID: \"c5b21a6e-2f4a-43ea-b85d-980b2169cea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:02:36.657939 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.657850    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5b21a6e-2f4a-43ea-b85d-980b2169cea1-maas-api-tls\") pod \"maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb\" (UID: \"c5b21a6e-2f4a-43ea-b85d-980b2169cea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:02:36.658044 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:02:36.657955    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-serving-cert: secret "maas-api-e2e-ait-3fc8989b-serving-cert" not found
Jul 17 13:02:36.658044 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:02:36.658038    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5b21a6e-2f4a-43ea-b85d-980b2169cea1-maas-api-tls podName:c5b21a6e-2f4a-43ea-b85d-980b2169cea1 nodeName:}" failed. No retries permitted until 2026-07-17 13:02:37.158021878 +0000 UTC m=+2401.289406516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c5b21a6e-2f4a-43ea-b85d-980b2169cea1-maas-api-tls") pod "maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" (UID: "c5b21a6e-2f4a-43ea-b85d-980b2169cea1") : secret "maas-api-e2e-ait-3fc8989b-serving-cert" not found
Jul 17 13:02:36.658461 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.658440    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5b21a6e-2f4a-43ea-b85d-980b2169cea1-openshift-service-ca\") pod \"maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb\" (UID: \"c5b21a6e-2f4a-43ea-b85d-980b2169cea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:02:36.667317 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.667301    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkhnq\" (UniqueName: \"kubernetes.io/projected/c5b21a6e-2f4a-43ea-b85d-980b2169cea1-kube-api-access-xkhnq\") pod \"maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb\" (UID: \"c5b21a6e-2f4a-43ea-b85d-980b2169cea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:02:36.824629 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:36.824588    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:02:37.126299 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:37.126214    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:02:37.162530 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:37.162504    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5b21a6e-2f4a-43ea-b85d-980b2169cea1-maas-api-tls\") pod \"maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb\" (UID: \"c5b21a6e-2f4a-43ea-b85d-980b2169cea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:02:37.165031 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:37.165011    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5b21a6e-2f4a-43ea-b85d-980b2169cea1-maas-api-tls\") pod \"maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb\" (UID: \"c5b21a6e-2f4a-43ea-b85d-980b2169cea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:02:37.234483 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:37.234452    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:02:37.455754 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:37.455682    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:02:37.598085 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:37.598056    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"]
Jul 17 13:02:37.599744 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:02:37.599718    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b21a6e_2f4a_43ea_b85d_980b2169cea1.slice/crio-8a5975ea084c282c908c73c238a1b7490611c92d9e5ef91b8e9d69885b8622cf WatchSource:0}: Error finding container 8a5975ea084c282c908c73c238a1b7490611c92d9e5ef91b8e9d69885b8622cf: Status 404 returned error can't find the container with id 8a5975ea084c282c908c73c238a1b7490611c92d9e5ef91b8e9d69885b8622cf
Jul 17 13:02:38.094434 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:38.094396    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" event={"ID":"c5b21a6e-2f4a-43ea-b85d-980b2169cea1","Type":"ContainerStarted","Data":"8a5975ea084c282c908c73c238a1b7490611c92d9e5ef91b8e9d69885b8622cf"}
Jul 17 13:02:38.879663 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:38.879634    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:02:39.101412 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:39.101342    2565 generic.go:356] "Generic (PLEG): container finished" podID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerID="649cf44776f256330747191c38dd5a1ab62cd07bec00f452e43bf459a8d37ee1" exitCode=1
Jul 17 13:02:39.101598 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:39.101437    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" event={"ID":"c5b21a6e-2f4a-43ea-b85d-980b2169cea1","Type":"ContainerDied","Data":"649cf44776f256330747191c38dd5a1ab62cd07bec00f452e43bf459a8d37ee1"}
Jul 17 13:02:39.101665 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:39.101626    2565 scope.go:117] "RemoveContainer" containerID="649cf44776f256330747191c38dd5a1ab62cd07bec00f452e43bf459a8d37ee1"
Jul 17 13:02:40.110455 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:40.110424    2565 generic.go:356] "Generic (PLEG): container finished" podID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerID="4d26b4deb3eea7f2d012d93da1dc304aa0c2add3aeca43646f89f005d603931d" exitCode=1
Jul 17 13:02:40.110912 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:40.110516    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" event={"ID":"c5b21a6e-2f4a-43ea-b85d-980b2169cea1","Type":"ContainerDied","Data":"4d26b4deb3eea7f2d012d93da1dc304aa0c2add3aeca43646f89f005d603931d"}
Jul 17 13:02:40.110912 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:40.110584    2565 scope.go:117] "RemoveContainer" containerID="649cf44776f256330747191c38dd5a1ab62cd07bec00f452e43bf459a8d37ee1"
Jul 17 13:02:40.110912 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:40.110754    2565 scope.go:117] "RemoveContainer" containerID="4d26b4deb3eea7f2d012d93da1dc304aa0c2add3aeca43646f89f005d603931d"
Jul 17 13:02:40.111114 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:02:40.110971    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb_odh-ai-gateway-infra(c5b21a6e-2f4a-43ea-b85d-980b2169cea1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1"
Jul 17 13:02:41.119265 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:41.119237    2565 scope.go:117] "RemoveContainer" containerID="4d26b4deb3eea7f2d012d93da1dc304aa0c2add3aeca43646f89f005d603931d"
Jul 17 13:02:41.119697 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:02:41.119450    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb_odh-ai-gateway-infra(c5b21a6e-2f4a-43ea-b85d-980b2169cea1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1"
Jul 17 13:02:42.456090 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:42.456045    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:02:42.456443 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:42.456397    2565 scope.go:117] "RemoveContainer" containerID="4d26b4deb3eea7f2d012d93da1dc304aa0c2add3aeca43646f89f005d603931d"
Jul 17 13:02:42.456576 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:02:42.456554    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb_odh-ai-gateway-infra(c5b21a6e-2f4a-43ea-b85d-980b2169cea1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1"
Jul 17 13:02:47.456686 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:47.456659    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:02:47.457166 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:47.457068    2565 scope.go:117] "RemoveContainer" containerID="4d26b4deb3eea7f2d012d93da1dc304aa0c2add3aeca43646f89f005d603931d"
Jul 17 13:02:47.457288 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:02:47.457266    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb_odh-ai-gateway-infra(c5b21a6e-2f4a-43ea-b85d-980b2169cea1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1"
Jul 17 13:02:59.396509 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:02:59.396443    2565 scope.go:117] "RemoveContainer" containerID="4d26b4deb3eea7f2d012d93da1dc304aa0c2add3aeca43646f89f005d603931d"
Jul 17 13:03:00.233948 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:03:00.233914    2565 generic.go:356] "Generic (PLEG): container finished" podID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerID="27d9c794f5b6b30707d0de40a540f947d209a214dad909dbb4db4f14fc6e0bb7" exitCode=1
Jul 17 13:03:00.234108 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:03:00.234004    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" event={"ID":"c5b21a6e-2f4a-43ea-b85d-980b2169cea1","Type":"ContainerDied","Data":"27d9c794f5b6b30707d0de40a540f947d209a214dad909dbb4db4f14fc6e0bb7"}
Jul 17 13:03:00.234108 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:03:00.234088    2565 scope.go:117] "RemoveContainer" containerID="4d26b4deb3eea7f2d012d93da1dc304aa0c2add3aeca43646f89f005d603931d"
Jul 17 13:03:00.234400 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:03:00.234381    2565 scope.go:117] "RemoveContainer" containerID="27d9c794f5b6b30707d0de40a540f947d209a214dad909dbb4db4f14fc6e0bb7"
Jul 17 13:03:00.234579 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:03:00.234560    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb_odh-ai-gateway-infra(c5b21a6e-2f4a-43ea-b85d-980b2169cea1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1"
Jul 17 13:03:02.456230 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:03:02.456191    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:03:02.456584 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:03:02.456502    2565 scope.go:117] "RemoveContainer" containerID="27d9c794f5b6b30707d0de40a540f947d209a214dad909dbb4db4f14fc6e0bb7"
Jul 17 13:03:02.456674 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:03:02.456657    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb_odh-ai-gateway-infra(c5b21a6e-2f4a-43ea-b85d-980b2169cea1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1"
Jul 17 13:03:07.456497 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:03:07.456461    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:03:07.456854 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:03:07.456806    2565 scope.go:117] "RemoveContainer" containerID="27d9c794f5b6b30707d0de40a540f947d209a214dad909dbb4db4f14fc6e0bb7"
Jul 17 13:03:07.456977 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:03:07.456956    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb_odh-ai-gateway-infra(c5b21a6e-2f4a-43ea-b85d-980b2169cea1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1"
Jul 17 13:03:22.396789 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:03:22.396759    2565 scope.go:117] "RemoveContainer" containerID="27d9c794f5b6b30707d0de40a540f947d209a214dad909dbb4db4f14fc6e0bb7"
Jul 17 13:03:23.372538 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:03:23.372508    2565 generic.go:356] "Generic (PLEG): container finished" podID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerID="eaf3d60bc59b41749a78235e896a0ea0794fec0cc9b96285696b099c0b5645c9" exitCode=1
Jul 17 13:03:23.372707 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:03:23.372582    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" event={"ID":"c5b21a6e-2f4a-43ea-b85d-980b2169cea1","Type":"ContainerDied","Data":"eaf3d60bc59b41749a78235e896a0ea0794fec0cc9b96285696b099c0b5645c9"}
Jul 17 13:03:23.372707 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:03:23.372676    2565 scope.go:117] "RemoveContainer" containerID="27d9c794f5b6b30707d0de40a540f947d209a214dad909dbb4db4f14fc6e0bb7"
Jul 17 13:03:23.373029 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:03:23.373010    2565 scope.go:117] "RemoveContainer" containerID="eaf3d60bc59b41749a78235e896a0ea0794fec0cc9b96285696b099c0b5645c9"
Jul 17 13:03:23.373191 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:03:23.373174    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb_odh-ai-gateway-infra(c5b21a6e-2f4a-43ea-b85d-980b2169cea1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1"
Jul 17 13:03:27.456796 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:03:27.456763    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:03:27.457185 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:03:27.456806    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:03:27.457185 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:03:27.457120    2565 scope.go:117] "RemoveContainer" containerID="eaf3d60bc59b41749a78235e896a0ea0794fec0cc9b96285696b099c0b5645c9"
Jul 17 13:03:27.457278 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:03:27.457258    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb_odh-ai-gateway-infra(c5b21a6e-2f4a-43ea-b85d-980b2169cea1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1"
Jul 17 13:03:41.396649 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:03:41.396623    2565 scope.go:117] "RemoveContainer" containerID="eaf3d60bc59b41749a78235e896a0ea0794fec0cc9b96285696b099c0b5645c9"
Jul 17 13:03:41.397124 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:03:41.396783    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb_odh-ai-gateway-infra(c5b21a6e-2f4a-43ea-b85d-980b2169cea1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1"
Jul 17 13:03:55.396826 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:03:55.396800    2565 scope.go:117] "RemoveContainer" containerID="eaf3d60bc59b41749a78235e896a0ea0794fec0cc9b96285696b099c0b5645c9"
Jul 17 13:03:55.397199 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:03:55.396957    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb_odh-ai-gateway-infra(c5b21a6e-2f4a-43ea-b85d-980b2169cea1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1"
Jul 17 13:04:03.934249 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:04:03.934216    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:04:04.035293 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:04:04.035261    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:04:10.396780 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:04:10.396755    2565 scope.go:117] "RemoveContainer" containerID="eaf3d60bc59b41749a78235e896a0ea0794fec0cc9b96285696b099c0b5645c9"
Jul 17 13:04:11.652239 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:04:11.652210    2565 generic.go:356] "Generic (PLEG): container finished" podID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerID="f1d121f159ffd8385c64cc79473e81ca9002b039049c77801be69c256edd65ce" exitCode=1
Jul 17 13:04:11.652653 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:04:11.652277    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" event={"ID":"c5b21a6e-2f4a-43ea-b85d-980b2169cea1","Type":"ContainerDied","Data":"f1d121f159ffd8385c64cc79473e81ca9002b039049c77801be69c256edd65ce"}
Jul 17 13:04:11.652653 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:04:11.652333    2565 scope.go:117] "RemoveContainer" containerID="eaf3d60bc59b41749a78235e896a0ea0794fec0cc9b96285696b099c0b5645c9"
Jul 17 13:04:11.652653 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:04:11.652578    2565 scope.go:117] "RemoveContainer" containerID="f1d121f159ffd8385c64cc79473e81ca9002b039049c77801be69c256edd65ce"
Jul 17 13:04:11.652773 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:04:11.652761    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb_odh-ai-gateway-infra(c5b21a6e-2f4a-43ea-b85d-980b2169cea1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1"
Jul 17 13:04:12.456205 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:04:12.456177    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:04:12.659506 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:04:12.659475    2565 scope.go:117] "RemoveContainer" containerID="f1d121f159ffd8385c64cc79473e81ca9002b039049c77801be69c256edd65ce"
Jul 17 13:04:12.659910 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:04:12.659641    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb_odh-ai-gateway-infra(c5b21a6e-2f4a-43ea-b85d-980b2169cea1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1"
Jul 17 13:04:17.456679 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:04:17.456649    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:04:17.457046 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:04:17.457010    2565 scope.go:117] "RemoveContainer" containerID="f1d121f159ffd8385c64cc79473e81ca9002b039049c77801be69c256edd65ce"
Jul 17 13:04:17.457182 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:04:17.457155    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb_odh-ai-gateway-infra(c5b21a6e-2f4a-43ea-b85d-980b2169cea1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1"
Jul 17 13:04:32.397657 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:04:32.397581    2565 scope.go:117] "RemoveContainer" containerID="f1d121f159ffd8385c64cc79473e81ca9002b039049c77801be69c256edd65ce"
Jul 17 13:04:32.398057 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:04:32.397859    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb_odh-ai-gateway-infra(c5b21a6e-2f4a-43ea-b85d-980b2169cea1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1"
Jul 17 13:04:43.397439 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:04:43.397400    2565 scope.go:117] "RemoveContainer" containerID="f1d121f159ffd8385c64cc79473e81ca9002b039049c77801be69c256edd65ce"
Jul 17 13:04:43.397833 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:04:43.397600    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb_odh-ai-gateway-infra(c5b21a6e-2f4a-43ea-b85d-980b2169cea1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1"
Jul 17 13:04:58.397401 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:04:58.397364    2565 scope.go:117] "RemoveContainer" containerID="f1d121f159ffd8385c64cc79473e81ca9002b039049c77801be69c256edd65ce"
Jul 17 13:04:58.399816 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:04:58.397612    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb_odh-ai-gateway-infra(c5b21a6e-2f4a-43ea-b85d-980b2169cea1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1"
Jul 17 13:05:13.397511 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:13.397481    2565 scope.go:117] "RemoveContainer" containerID="f1d121f159ffd8385c64cc79473e81ca9002b039049c77801be69c256edd65ce"
Jul 17 13:05:13.397917 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:05:13.397684    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb_odh-ai-gateway-infra(c5b21a6e-2f4a-43ea-b85d-980b2169cea1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1"
Jul 17 13:05:27.396703 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:27.396673    2565 scope.go:117] "RemoveContainer" containerID="f1d121f159ffd8385c64cc79473e81ca9002b039049c77801be69c256edd65ce"
Jul 17 13:05:27.397180 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:05:27.396852    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb_odh-ai-gateway-infra(c5b21a6e-2f4a-43ea-b85d-980b2169cea1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1"
Jul 17 13:05:41.396733 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:41.396702    2565 scope.go:117] "RemoveContainer" containerID="f1d121f159ffd8385c64cc79473e81ca9002b039049c77801be69c256edd65ce"
Jul 17 13:05:41.397183 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:41.397073    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 13:05:41.460585 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:41.460556    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:05:42.177498 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:42.177469    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" event={"ID":"c5b21a6e-2f4a-43ea-b85d-980b2169cea1","Type":"ContainerStarted","Data":"4152b1627e932226a8aea48df440f283301b712005eb466eb63878fd9226cec9"}
Jul 17 13:05:42.177707 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:42.177690    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:05:42.200155 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:42.200096    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podStartSLOduration=183.824822594 podStartE2EDuration="3m6.200083132s" totalImagesPullingTime="2.375260538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:02:36 +0000 UTC" imagePullSessionsCount=6 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:05:42.197419957 +0000 UTC m=+2586.328804600" watchObservedRunningTime="2026-07-17 13:05:42.200083132 +0000 UTC m=+2586.331467793"
Jul 17 13:05:43.185784 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:43.185756    2565 generic.go:356] "Generic (PLEG): container finished" podID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerID="4152b1627e932226a8aea48df440f283301b712005eb466eb63878fd9226cec9" exitCode=1
Jul 17 13:05:43.186198 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:43.185839    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" event={"ID":"c5b21a6e-2f4a-43ea-b85d-980b2169cea1","Type":"ContainerDied","Data":"4152b1627e932226a8aea48df440f283301b712005eb466eb63878fd9226cec9"}
Jul 17 13:05:43.186198 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:43.185906    2565 scope.go:117] "RemoveContainer" containerID="f1d121f159ffd8385c64cc79473e81ca9002b039049c77801be69c256edd65ce"
Jul 17 13:05:43.186198 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:43.186152    2565 scope.go:117] "RemoveContainer" containerID="4152b1627e932226a8aea48df440f283301b712005eb466eb63878fd9226cec9"
Jul 17 13:05:43.186415 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:05:43.186398    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb_odh-ai-gateway-infra(c5b21a6e-2f4a-43ea-b85d-980b2169cea1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1"
Jul 17 13:05:44.193400 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:44.193365    2565 scope.go:117] "RemoveContainer" containerID="4152b1627e932226a8aea48df440f283301b712005eb466eb63878fd9226cec9"
Jul 17 13:05:44.193795 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:05:44.193544    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb_odh-ai-gateway-infra(c5b21a6e-2f4a-43ea-b85d-980b2169cea1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1"
Jul 17 13:05:44.824545 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:44.824513    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:05:47.455785 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:47.455758    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:05:47.456169 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:47.456152    2565 scope.go:117] "RemoveContainer" containerID="4152b1627e932226a8aea48df440f283301b712005eb466eb63878fd9226cec9"
Jul 17 13:05:47.456325 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:05:47.456305    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb_odh-ai-gateway-infra(c5b21a6e-2f4a-43ea-b85d-980b2169cea1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1"
Jul 17 13:05:47.802635 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:47.802606    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"]
Jul 17 13:05:47.838140 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:47.838109    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:05:48.038547 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:48.038513    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:05:48.355309 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:48.355286    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:05:48.463833 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:48.463804    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkhnq\" (UniqueName: \"kubernetes.io/projected/c5b21a6e-2f4a-43ea-b85d-980b2169cea1-kube-api-access-xkhnq\") pod \"c5b21a6e-2f4a-43ea-b85d-980b2169cea1\" (UID: \"c5b21a6e-2f4a-43ea-b85d-980b2169cea1\") "
Jul 17 13:05:48.464293 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:48.463861    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5b21a6e-2f4a-43ea-b85d-980b2169cea1-maas-api-tls\") pod \"c5b21a6e-2f4a-43ea-b85d-980b2169cea1\" (UID: \"c5b21a6e-2f4a-43ea-b85d-980b2169cea1\") "
Jul 17 13:05:48.464293 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:48.463881    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5b21a6e-2f4a-43ea-b85d-980b2169cea1-openshift-service-ca\") pod \"c5b21a6e-2f4a-43ea-b85d-980b2169cea1\" (UID: \"c5b21a6e-2f4a-43ea-b85d-980b2169cea1\") "
Jul 17 13:05:48.464386 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:48.464315    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5b21a6e-2f4a-43ea-b85d-980b2169cea1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c5b21a6e-2f4a-43ea-b85d-980b2169cea1" (UID: "c5b21a6e-2f4a-43ea-b85d-980b2169cea1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:05:48.466315 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:48.466287    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b21a6e-2f4a-43ea-b85d-980b2169cea1-kube-api-access-xkhnq" (OuterVolumeSpecName: "kube-api-access-xkhnq") pod "c5b21a6e-2f4a-43ea-b85d-980b2169cea1" (UID: "c5b21a6e-2f4a-43ea-b85d-980b2169cea1"). InnerVolumeSpecName "kube-api-access-xkhnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:05:48.466421 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:48.466336    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5b21a6e-2f4a-43ea-b85d-980b2169cea1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c5b21a6e-2f4a-43ea-b85d-980b2169cea1" (UID: "c5b21a6e-2f4a-43ea-b85d-980b2169cea1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 13:05:48.564687 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:48.564654    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkhnq\" (UniqueName: \"kubernetes.io/projected/c5b21a6e-2f4a-43ea-b85d-980b2169cea1-kube-api-access-xkhnq\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:05:48.564687 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:48.564682    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5b21a6e-2f4a-43ea-b85d-980b2169cea1-maas-api-tls\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:05:48.564687 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:48.564691    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5b21a6e-2f4a-43ea-b85d-980b2169cea1-openshift-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:05:49.223575 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:49.223543    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb" event={"ID":"c5b21a6e-2f4a-43ea-b85d-980b2169cea1","Type":"ContainerDied","Data":"8a5975ea084c282c908c73c238a1b7490611c92d9e5ef91b8e9d69885b8622cf"}
Jul 17 13:05:49.223755 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:49.223599    2565 scope.go:117] "RemoveContainer" containerID="4152b1627e932226a8aea48df440f283301b712005eb466eb63878fd9226cec9"
Jul 17 13:05:49.223755 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:49.223556    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"
Jul 17 13:05:49.257846 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:49.257824    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"]
Jul 17 13:05:49.268219 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:49.268194    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3fc8989b-7b8d5dbb6d-8mglb"]
Jul 17 13:05:50.408871 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:50.408836    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" path="/var/lib/kubelet/pods/c5b21a6e-2f4a-43ea-b85d-980b2169cea1/volumes"
Jul 17 13:05:53.935951 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:53.935913    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:05:54.035637 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:54.035603    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:05:54.236470 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:54.236376    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:05:54.358892 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:54.358858    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:05:54.512739 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:54.512648    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:05:55.708766 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:55.708727    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:05:55.930372 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:55.930344    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:05:56.104800 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:56.104771    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:05:56.447091 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:56.447016    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:05:56.550028 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:05:56.549970    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:06:02.967386 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:02.967347    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:06:03.033324 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:03.033275    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:06:07.378712 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:07.378668    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:06:07.480342 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:07.480310    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:06:17.369067 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:17.369030    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:06:17.466661 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:17.466632    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:06:31.450916 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:31.450877    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8"]
Jul 17 13:06:31.451336 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:31.451322    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerName="maas-api"
Jul 17 13:06:31.451381 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:31.451337    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerName="maas-api"
Jul 17 13:06:31.451381 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:31.451345    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerName="maas-api"
Jul 17 13:06:31.451381 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:31.451350    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerName="maas-api"
Jul 17 13:06:31.451381 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:31.451371    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerName="maas-api"
Jul 17 13:06:31.451381 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:31.451377    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerName="maas-api"
Jul 17 13:06:31.451545 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:31.451386    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerName="maas-api"
Jul 17 13:06:31.451545 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:31.451391    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerName="maas-api"
Jul 17 13:06:31.451545 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:31.451453    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerName="maas-api"
Jul 17 13:06:31.451545 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:31.451460    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerName="maas-api"
Jul 17 13:06:31.451545 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:31.451469    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerName="maas-api"
Jul 17 13:06:31.451545 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:31.451475    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerName="maas-api"
Jul 17 13:06:31.451545 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:31.451480    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerName="maas-api"
Jul 17 13:06:31.454686 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:31.454671    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8"
Jul 17 13:06:31.458008 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:31.457968    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:06:31.462514 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:31.462481    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8"]
Jul 17 13:06:31.561195 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:31.561154    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c643156-8044-4d25-b428-f56965720e58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8\" (UID: \"9c643156-8044-4d25-b428-f56965720e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8"
Jul 17 13:06:31.662286 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:31.662247    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c643156-8044-4d25-b428-f56965720e58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8\" (UID: \"9c643156-8044-4d25-b428-f56965720e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8"
Jul 17 13:06:31.662851 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:31.662831    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c643156-8044-4d25-b428-f56965720e58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8\" (UID: \"9c643156-8044-4d25-b428-f56965720e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8"
Jul 17 13:06:31.767828 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:31.767737    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8"
Jul 17 13:06:31.914938 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:31.914910    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8"]
Jul 17 13:06:31.915737 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:06:31.915712    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c643156_8044_4d25_b428_f56965720e58.slice/crio-ab438a24c76e584648e0689c716e583f1494a07de2a04f8e2d1d108d3b6f11cc WatchSource:0}: Error finding container ab438a24c76e584648e0689c716e583f1494a07de2a04f8e2d1d108d3b6f11cc: Status 404 returned error can't find the container with id ab438a24c76e584648e0689c716e583f1494a07de2a04f8e2d1d108d3b6f11cc
Jul 17 13:06:32.482249 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:32.482215    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8" event={"ID":"9c643156-8044-4d25-b428-f56965720e58","Type":"ContainerStarted","Data":"a20c141fc88be8d1c74f15e2f3cf85cf4060063898c264f65fcfc97d68de8de4"}
Jul 17 13:06:32.482648 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:32.482274    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8" event={"ID":"9c643156-8044-4d25-b428-f56965720e58","Type":"ContainerStarted","Data":"ab438a24c76e584648e0689c716e583f1494a07de2a04f8e2d1d108d3b6f11cc"}
Jul 17 13:06:32.501947 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:32.501895    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8" podStartSLOduration=1.501877458 podStartE2EDuration="1.501877458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:06:32.500212758 +0000 UTC m=+2636.631597437" watchObservedRunningTime="2026-07-17 13:06:32.501877458 +0000 UTC m=+2636.633262123"
Jul 17 13:06:33.489736 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:33.489695    2565 generic.go:356] "Generic (PLEG): container finished" podID="9c643156-8044-4d25-b428-f56965720e58" containerID="a20c141fc88be8d1c74f15e2f3cf85cf4060063898c264f65fcfc97d68de8de4" exitCode=6
Jul 17 13:06:33.490122 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:33.489780    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8" event={"ID":"9c643156-8044-4d25-b428-f56965720e58","Type":"ContainerDied","Data":"a20c141fc88be8d1c74f15e2f3cf85cf4060063898c264f65fcfc97d68de8de4"}
Jul 17 13:06:33.490122 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:33.490026    2565 scope.go:117] "RemoveContainer" containerID="a20c141fc88be8d1c74f15e2f3cf85cf4060063898c264f65fcfc97d68de8de4"
Jul 17 13:06:34.048585 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:34.048546    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:06:34.497248 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:34.497144    2565 generic.go:356] "Generic (PLEG): container finished" podID="9c643156-8044-4d25-b428-f56965720e58" containerID="890e4def6bfc88c8a97cc9e730740dc07305d506b946a1a32e9bcbfa0805cc50" exitCode=6
Jul 17 13:06:34.497248 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:34.497229    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8" event={"ID":"9c643156-8044-4d25-b428-f56965720e58","Type":"ContainerDied","Data":"890e4def6bfc88c8a97cc9e730740dc07305d506b946a1a32e9bcbfa0805cc50"}
Jul 17 13:06:34.497757 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:34.497295    2565 scope.go:117] "RemoveContainer" containerID="a20c141fc88be8d1c74f15e2f3cf85cf4060063898c264f65fcfc97d68de8de4"
Jul 17 13:06:34.497757 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:34.497458    2565 scope.go:117] "RemoveContainer" containerID="890e4def6bfc88c8a97cc9e730740dc07305d506b946a1a32e9bcbfa0805cc50"
Jul 17 13:06:34.497757 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:06:34.497741    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8_odh-ai-gateway-infra(9c643156-8044-4d25-b428-f56965720e58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8" podUID="9c643156-8044-4d25-b428-f56965720e58"
Jul 17 13:06:35.506500 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:35.506477    2565 scope.go:117] "RemoveContainer" containerID="890e4def6bfc88c8a97cc9e730740dc07305d506b946a1a32e9bcbfa0805cc50"
Jul 17 13:06:35.506880 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:06:35.506669    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8_odh-ai-gateway-infra(9c643156-8044-4d25-b428-f56965720e58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8" podUID="9c643156-8044-4d25-b428-f56965720e58"
Jul 17 13:06:44.961109 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:44.961062    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:06:45.025891 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:45.025851    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:06:49.397315 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:49.397287    2565 scope.go:117] "RemoveContainer" containerID="890e4def6bfc88c8a97cc9e730740dc07305d506b946a1a32e9bcbfa0805cc50"
Jul 17 13:06:50.598811 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:50.598778    2565 generic.go:356] "Generic (PLEG): container finished" podID="9c643156-8044-4d25-b428-f56965720e58" containerID="49dacb4a9ecdd3f3ee47a5944aa52fdea9fc4007e13b01ad3086785910f350d6" exitCode=6
Jul 17 13:06:50.599254 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:50.598829    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8" event={"ID":"9c643156-8044-4d25-b428-f56965720e58","Type":"ContainerDied","Data":"49dacb4a9ecdd3f3ee47a5944aa52fdea9fc4007e13b01ad3086785910f350d6"}
Jul 17 13:06:50.599254 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:50.598881    2565 scope.go:117] "RemoveContainer" containerID="890e4def6bfc88c8a97cc9e730740dc07305d506b946a1a32e9bcbfa0805cc50"
Jul 17 13:06:50.599254 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:50.599203    2565 scope.go:117] "RemoveContainer" containerID="49dacb4a9ecdd3f3ee47a5944aa52fdea9fc4007e13b01ad3086785910f350d6"
Jul 17 13:06:50.599461 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:06:50.599437    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8_odh-ai-gateway-infra(9c643156-8044-4d25-b428-f56965720e58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8" podUID="9c643156-8044-4d25-b428-f56965720e58"
Jul 17 13:06:51.628646 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:51.628620    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8"]
Jul 17 13:06:51.771266 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:51.771243    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8"
Jul 17 13:06:51.846190 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:51.846163    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c643156-8044-4d25-b428-f56965720e58-maas-api-service-ca\") pod \"9c643156-8044-4d25-b428-f56965720e58\" (UID: \"9c643156-8044-4d25-b428-f56965720e58\") "
Jul 17 13:06:51.846479 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:51.846458    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c643156-8044-4d25-b428-f56965720e58-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9c643156-8044-4d25-b428-f56965720e58" (UID: "9c643156-8044-4d25-b428-f56965720e58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:06:51.947449 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:51.947361    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c643156-8044-4d25-b428-f56965720e58-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:06:52.614045 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:52.614010    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8" event={"ID":"9c643156-8044-4d25-b428-f56965720e58","Type":"ContainerDied","Data":"ab438a24c76e584648e0689c716e583f1494a07de2a04f8e2d1d108d3b6f11cc"}
Jul 17 13:06:52.614045 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:52.614044    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8"
Jul 17 13:06:52.614308 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:52.614072    2565 scope.go:117] "RemoveContainer" containerID="49dacb4a9ecdd3f3ee47a5944aa52fdea9fc4007e13b01ad3086785910f350d6"
Jul 17 13:06:52.632490 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:52.632457    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8"]
Jul 17 13:06:52.635935 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:52.635904    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-5rlm8"]
Jul 17 13:06:54.403356 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:06:54.403323    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c643156-8044-4d25-b428-f56965720e58" path="/var/lib/kubelet/pods/9c643156-8044-4d25-b428-f56965720e58/volumes"
Jul 17 13:07:00.396489 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:07:00.396451    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:07:00.483045 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:07:00.482963    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:07:05.552061 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:07:05.552030    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:07:06.696945 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:07:06.696911    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:07:06.816501 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:07:06.816463    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:07:12.635272 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:07:12.635235    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:07:12.731751 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:07:12.731719    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:07:12.912639 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:07:12.912555    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:07:13.012847 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:07:13.012808    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:07:14.108900 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:07:14.108863    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:07:14.208947 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:07:14.208914    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:07:14.568507 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:07:14.568474    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:07:15.103041 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:07:15.103010    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:07:15.206366 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:07:15.206332    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:07:15.304492 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:07:15.304451    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:07:16.327252 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:07:16.327189    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:07:25.760840 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:07:25.760803    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:07:25.861946 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:07:25.861912    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:07:36.596866 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:07:36.596791    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 13:07:36.605458 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:07:36.605424    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 13:07:36.612594 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:07:36.612576    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 13:07:36.616962 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:07:36.616938    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 13:08:21.561534 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:21.561498    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft"]
Jul 17 13:08:21.562096 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:21.561946    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c643156-8044-4d25-b428-f56965720e58" containerName="revoke-keys"
Jul 17 13:08:21.562096 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:21.561962    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c643156-8044-4d25-b428-f56965720e58" containerName="revoke-keys"
Jul 17 13:08:21.562096 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:21.561998    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerName="maas-api"
Jul 17 13:08:21.562096 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:21.562003    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerName="maas-api"
Jul 17 13:08:21.562096 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:21.562010    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerName="maas-api"
Jul 17 13:08:21.562096 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:21.562019    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerName="maas-api"
Jul 17 13:08:21.562096 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:21.562040    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c643156-8044-4d25-b428-f56965720e58" containerName="revoke-keys"
Jul 17 13:08:21.562096 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:21.562046    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c643156-8044-4d25-b428-f56965720e58" containerName="revoke-keys"
Jul 17 13:08:21.562505 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:21.562117    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c643156-8044-4d25-b428-f56965720e58" containerName="revoke-keys"
Jul 17 13:08:21.562505 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:21.562126    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c643156-8044-4d25-b428-f56965720e58" containerName="revoke-keys"
Jul 17 13:08:21.562505 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:21.562133    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c643156-8044-4d25-b428-f56965720e58" containerName="revoke-keys"
Jul 17 13:08:21.562505 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:21.562143    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5b21a6e-2f4a-43ea-b85d-980b2169cea1" containerName="maas-api"
Jul 17 13:08:21.565308 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:21.565292    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft"
Jul 17 13:08:21.568195 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:21.568177    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:08:21.574726 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:21.574660    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft"]
Jul 17 13:08:21.680241 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:21.680198    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d00aeb-849b-4dcb-9309-caa63933b365-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft\" (UID: \"86d00aeb-849b-4dcb-9309-caa63933b365\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft"
Jul 17 13:08:21.780787 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:21.780752    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d00aeb-849b-4dcb-9309-caa63933b365-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft\" (UID: \"86d00aeb-849b-4dcb-9309-caa63933b365\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft"
Jul 17 13:08:21.781380 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:21.781361    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d00aeb-849b-4dcb-9309-caa63933b365-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft\" (UID: \"86d00aeb-849b-4dcb-9309-caa63933b365\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft"
Jul 17 13:08:21.877641 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:21.877554    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft"
Jul 17 13:08:22.229680 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:22.229646    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft"]
Jul 17 13:08:22.231820 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:08:22.231755    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d00aeb_849b_4dcb_9309_caa63933b365.slice/crio-7e9233efd85bbc4ffc826081ad9d91998344610d7a4bbbb33578bf41b2c111ac WatchSource:0}: Error finding container 7e9233efd85bbc4ffc826081ad9d91998344610d7a4bbbb33578bf41b2c111ac: Status 404 returned error can't find the container with id 7e9233efd85bbc4ffc826081ad9d91998344610d7a4bbbb33578bf41b2c111ac
Jul 17 13:08:23.147220 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:23.147134    2565 generic.go:356] "Generic (PLEG): container finished" podID="86d00aeb-849b-4dcb-9309-caa63933b365" containerID="b729832241918c5b1a5b8dfb84e1e619648875be3c129753eda5a93c15ee91b7" exitCode=6
Jul 17 13:08:23.147601 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:23.147259    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft" event={"ID":"86d00aeb-849b-4dcb-9309-caa63933b365","Type":"ContainerDied","Data":"b729832241918c5b1a5b8dfb84e1e619648875be3c129753eda5a93c15ee91b7"}
Jul 17 13:08:23.147601 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:23.147313    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft" event={"ID":"86d00aeb-849b-4dcb-9309-caa63933b365","Type":"ContainerStarted","Data":"7e9233efd85bbc4ffc826081ad9d91998344610d7a4bbbb33578bf41b2c111ac"}
Jul 17 13:08:23.147601 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:23.147428    2565 scope.go:117] "RemoveContainer" containerID="b729832241918c5b1a5b8dfb84e1e619648875be3c129753eda5a93c15ee91b7"
Jul 17 13:08:24.155506 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:24.155414    2565 generic.go:356] "Generic (PLEG): container finished" podID="86d00aeb-849b-4dcb-9309-caa63933b365" containerID="493bab6cd3287d6de99d6df76baf64d4f612bad83f86b55aa83b69570627ede5" exitCode=6
Jul 17 13:08:24.155960 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:24.155502    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft" event={"ID":"86d00aeb-849b-4dcb-9309-caa63933b365","Type":"ContainerDied","Data":"493bab6cd3287d6de99d6df76baf64d4f612bad83f86b55aa83b69570627ede5"}
Jul 17 13:08:24.155960 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:24.155585    2565 scope.go:117] "RemoveContainer" containerID="b729832241918c5b1a5b8dfb84e1e619648875be3c129753eda5a93c15ee91b7"
Jul 17 13:08:24.155960 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:24.155774    2565 scope.go:117] "RemoveContainer" containerID="493bab6cd3287d6de99d6df76baf64d4f612bad83f86b55aa83b69570627ede5"
Jul 17 13:08:24.156157 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:08:24.156037    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft_odh-ai-gateway-infra(86d00aeb-849b-4dcb-9309-caa63933b365)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft" podUID="86d00aeb-849b-4dcb-9309-caa63933b365"
Jul 17 13:08:25.163793 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:25.163759    2565 scope.go:117] "RemoveContainer" containerID="493bab6cd3287d6de99d6df76baf64d4f612bad83f86b55aa83b69570627ede5"
Jul 17 13:08:25.164197 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:08:25.164029    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft_odh-ai-gateway-infra(86d00aeb-849b-4dcb-9309-caa63933b365)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft" podUID="86d00aeb-849b-4dcb-9309-caa63933b365"
Jul 17 13:08:36.399897 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:36.399867    2565 scope.go:117] "RemoveContainer" containerID="493bab6cd3287d6de99d6df76baf64d4f612bad83f86b55aa83b69570627ede5"
Jul 17 13:08:37.234572 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:37.234534    2565 generic.go:356] "Generic (PLEG): container finished" podID="86d00aeb-849b-4dcb-9309-caa63933b365" containerID="8e92a82609bef523e0947c4a41171b62bbbb7e9219dd977404ef8ee35c3957c1" exitCode=6
Jul 17 13:08:37.234572 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:37.234561    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft" event={"ID":"86d00aeb-849b-4dcb-9309-caa63933b365","Type":"ContainerDied","Data":"8e92a82609bef523e0947c4a41171b62bbbb7e9219dd977404ef8ee35c3957c1"}
Jul 17 13:08:37.234768 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:37.234622    2565 scope.go:117] "RemoveContainer" containerID="493bab6cd3287d6de99d6df76baf64d4f612bad83f86b55aa83b69570627ede5"
Jul 17 13:08:37.234905 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:37.234883    2565 scope.go:117] "RemoveContainer" containerID="8e92a82609bef523e0947c4a41171b62bbbb7e9219dd977404ef8ee35c3957c1"
Jul 17 13:08:37.235132 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:08:37.235112    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft_odh-ai-gateway-infra(86d00aeb-849b-4dcb-9309-caa63933b365)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft" podUID="86d00aeb-849b-4dcb-9309-caa63933b365"
Jul 17 13:08:38.271897 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:38.271864    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft"]
Jul 17 13:08:38.422994 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:38.422965    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft"
Jul 17 13:08:38.438703 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:38.438680    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d00aeb-849b-4dcb-9309-caa63933b365-maas-api-service-ca\") pod \"86d00aeb-849b-4dcb-9309-caa63933b365\" (UID: \"86d00aeb-849b-4dcb-9309-caa63933b365\") "
Jul 17 13:08:38.439004 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:38.438965    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d00aeb-849b-4dcb-9309-caa63933b365-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86d00aeb-849b-4dcb-9309-caa63933b365" (UID: "86d00aeb-849b-4dcb-9309-caa63933b365"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:08:38.439111 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:38.439056    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d00aeb-849b-4dcb-9309-caa63933b365-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:08:39.248724 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:39.248696    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft"
Jul 17 13:08:39.248724 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:39.248717    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft" event={"ID":"86d00aeb-849b-4dcb-9309-caa63933b365","Type":"ContainerDied","Data":"7e9233efd85bbc4ffc826081ad9d91998344610d7a4bbbb33578bf41b2c111ac"}
Jul 17 13:08:39.248958 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:39.248769    2565 scope.go:117] "RemoveContainer" containerID="8e92a82609bef523e0947c4a41171b62bbbb7e9219dd977404ef8ee35c3957c1"
Jul 17 13:08:39.274311 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:39.274280    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft"]
Jul 17 13:08:39.283101 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:39.283079    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-ht4ft"]
Jul 17 13:08:40.402842 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:08:40.402811    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86d00aeb-849b-4dcb-9309-caa63933b365" path="/var/lib/kubelet/pods/86d00aeb-849b-4dcb-9309-caa63933b365/volumes"
Jul 17 13:09:11.605206 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:11.605129    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx"]
Jul 17 13:09:11.605794 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:11.605532    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86d00aeb-849b-4dcb-9309-caa63933b365" containerName="revoke-keys"
Jul 17 13:09:11.605794 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:11.605543    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="86d00aeb-849b-4dcb-9309-caa63933b365" containerName="revoke-keys"
Jul 17 13:09:11.605794 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:11.605559    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86d00aeb-849b-4dcb-9309-caa63933b365" containerName="revoke-keys"
Jul 17 13:09:11.605794 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:11.605564    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="86d00aeb-849b-4dcb-9309-caa63933b365" containerName="revoke-keys"
Jul 17 13:09:11.605794 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:11.605577    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c643156-8044-4d25-b428-f56965720e58" containerName="revoke-keys"
Jul 17 13:09:11.605794 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:11.605582    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c643156-8044-4d25-b428-f56965720e58" containerName="revoke-keys"
Jul 17 13:09:11.605794 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:11.605666    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="86d00aeb-849b-4dcb-9309-caa63933b365" containerName="revoke-keys"
Jul 17 13:09:11.605794 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:11.605673    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="86d00aeb-849b-4dcb-9309-caa63933b365" containerName="revoke-keys"
Jul 17 13:09:11.605794 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:11.605680    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="86d00aeb-849b-4dcb-9309-caa63933b365" containerName="revoke-keys"
Jul 17 13:09:11.608897 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:11.608880    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx"
Jul 17 13:09:11.611913 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:11.611894    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:09:11.615976 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:11.615951    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx"]
Jul 17 13:09:11.697075 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:11.697051    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43ead2d-6107-479a-a24a-f4d60ef818d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx\" (UID: \"c43ead2d-6107-479a-a24a-f4d60ef818d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx"
Jul 17 13:09:11.797604 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:11.797569    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43ead2d-6107-479a-a24a-f4d60ef818d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx\" (UID: \"c43ead2d-6107-479a-a24a-f4d60ef818d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx"
Jul 17 13:09:11.798180 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:11.798159    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43ead2d-6107-479a-a24a-f4d60ef818d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx\" (UID: \"c43ead2d-6107-479a-a24a-f4d60ef818d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx"
Jul 17 13:09:11.920902 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:11.920840    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx"
Jul 17 13:09:12.065616 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:12.065591    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx"]
Jul 17 13:09:12.067395 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:09:12.067360    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43ead2d_6107_479a_a24a_f4d60ef818d9.slice/crio-8f7e604b4501c4ae33c1fd61a2051aaf03b508881ac80a135f76c7e0f5913912 WatchSource:0}: Error finding container 8f7e604b4501c4ae33c1fd61a2051aaf03b508881ac80a135f76c7e0f5913912: Status 404 returned error can't find the container with id 8f7e604b4501c4ae33c1fd61a2051aaf03b508881ac80a135f76c7e0f5913912
Jul 17 13:09:12.452316 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:12.452286    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx" event={"ID":"c43ead2d-6107-479a-a24a-f4d60ef818d9","Type":"ContainerStarted","Data":"962014c3d2766c11cf28bfc91ae374b8ac8364e48c684067489221381f0d9bc7"}
Jul 17 13:09:12.452432 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:12.452327    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx" event={"ID":"c43ead2d-6107-479a-a24a-f4d60ef818d9","Type":"ContainerStarted","Data":"8f7e604b4501c4ae33c1fd61a2051aaf03b508881ac80a135f76c7e0f5913912"}
Jul 17 13:09:12.472922 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:12.472846    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx" podStartSLOduration=1.47283089 podStartE2EDuration="1.47283089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:09:12.470229434 +0000 UTC m=+2796.601614094" watchObservedRunningTime="2026-07-17 13:09:12.47283089 +0000 UTC m=+2796.604215551"
Jul 17 13:09:13.459473 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:13.459443    2565 generic.go:356] "Generic (PLEG): container finished" podID="c43ead2d-6107-479a-a24a-f4d60ef818d9" containerID="962014c3d2766c11cf28bfc91ae374b8ac8364e48c684067489221381f0d9bc7" exitCode=6
Jul 17 13:09:13.459941 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:13.459530    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx" event={"ID":"c43ead2d-6107-479a-a24a-f4d60ef818d9","Type":"ContainerDied","Data":"962014c3d2766c11cf28bfc91ae374b8ac8364e48c684067489221381f0d9bc7"}
Jul 17 13:09:13.459941 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:13.459684    2565 scope.go:117] "RemoveContainer" containerID="962014c3d2766c11cf28bfc91ae374b8ac8364e48c684067489221381f0d9bc7"
Jul 17 13:09:14.467211 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:14.467174    2565 generic.go:356] "Generic (PLEG): container finished" podID="c43ead2d-6107-479a-a24a-f4d60ef818d9" containerID="4c9a012f3bc9e88075ecd87fa7344293f488399b5681edad86fcb2990dc6437e" exitCode=6
Jul 17 13:09:14.467661 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:14.467274    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx" event={"ID":"c43ead2d-6107-479a-a24a-f4d60ef818d9","Type":"ContainerDied","Data":"4c9a012f3bc9e88075ecd87fa7344293f488399b5681edad86fcb2990dc6437e"}
Jul 17 13:09:14.467661 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:14.467313    2565 scope.go:117] "RemoveContainer" containerID="962014c3d2766c11cf28bfc91ae374b8ac8364e48c684067489221381f0d9bc7"
Jul 17 13:09:14.467661 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:14.467447    2565 scope.go:117] "RemoveContainer" containerID="4c9a012f3bc9e88075ecd87fa7344293f488399b5681edad86fcb2990dc6437e"
Jul 17 13:09:14.467827 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:09:14.467668    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx_odh-ai-gateway-infra(c43ead2d-6107-479a-a24a-f4d60ef818d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx" podUID="c43ead2d-6107-479a-a24a-f4d60ef818d9"
Jul 17 13:09:15.475242 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:15.475217    2565 scope.go:117] "RemoveContainer" containerID="4c9a012f3bc9e88075ecd87fa7344293f488399b5681edad86fcb2990dc6437e"
Jul 17 13:09:15.475606 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:09:15.475370    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx_odh-ai-gateway-infra(c43ead2d-6107-479a-a24a-f4d60ef818d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx" podUID="c43ead2d-6107-479a-a24a-f4d60ef818d9"
Jul 17 13:09:26.400043 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:26.400015    2565 scope.go:117] "RemoveContainer" containerID="4c9a012f3bc9e88075ecd87fa7344293f488399b5681edad86fcb2990dc6437e"
Jul 17 13:09:27.546200 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:27.546123    2565 generic.go:356] "Generic (PLEG): container finished" podID="c43ead2d-6107-479a-a24a-f4d60ef818d9" containerID="9433b0b954fdd91d2e2bacea4ded5b0eff82d450ab6670fda32e6f706b1dbaa0" exitCode=6
Jul 17 13:09:27.546619 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:27.546203    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx" event={"ID":"c43ead2d-6107-479a-a24a-f4d60ef818d9","Type":"ContainerDied","Data":"9433b0b954fdd91d2e2bacea4ded5b0eff82d450ab6670fda32e6f706b1dbaa0"}
Jul 17 13:09:27.546619 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:27.546266    2565 scope.go:117] "RemoveContainer" containerID="4c9a012f3bc9e88075ecd87fa7344293f488399b5681edad86fcb2990dc6437e"
Jul 17 13:09:27.546619 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:27.546537    2565 scope.go:117] "RemoveContainer" containerID="9433b0b954fdd91d2e2bacea4ded5b0eff82d450ab6670fda32e6f706b1dbaa0"
Jul 17 13:09:27.546786 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:09:27.546733    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx_odh-ai-gateway-infra(c43ead2d-6107-479a-a24a-f4d60ef818d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx" podUID="c43ead2d-6107-479a-a24a-f4d60ef818d9"
Jul 17 13:09:28.578078 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:28.578047    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx"]
Jul 17 13:09:28.723067 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:28.723045    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx"
Jul 17 13:09:28.831791 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:28.831718    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43ead2d-6107-479a-a24a-f4d60ef818d9-maas-api-service-ca\") pod \"c43ead2d-6107-479a-a24a-f4d60ef818d9\" (UID: \"c43ead2d-6107-479a-a24a-f4d60ef818d9\") "
Jul 17 13:09:28.832033 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:28.832014    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c43ead2d-6107-479a-a24a-f4d60ef818d9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c43ead2d-6107-479a-a24a-f4d60ef818d9" (UID: "c43ead2d-6107-479a-a24a-f4d60ef818d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:09:28.932878 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:28.932850    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43ead2d-6107-479a-a24a-f4d60ef818d9-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:09:29.559707 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:29.559679    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx"
Jul 17 13:09:29.559883 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:29.559726    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx" event={"ID":"c43ead2d-6107-479a-a24a-f4d60ef818d9","Type":"ContainerDied","Data":"8f7e604b4501c4ae33c1fd61a2051aaf03b508881ac80a135f76c7e0f5913912"}
Jul 17 13:09:29.559883 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:29.559783    2565 scope.go:117] "RemoveContainer" containerID="9433b0b954fdd91d2e2bacea4ded5b0eff82d450ab6670fda32e6f706b1dbaa0"
Jul 17 13:09:29.584097 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:29.584070    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx"]
Jul 17 13:09:29.590477 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:29.590457    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-pswbx"]
Jul 17 13:09:30.403136 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:30.403107    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c43ead2d-6107-479a-a24a-f4d60ef818d9" path="/var/lib/kubelet/pods/c43ead2d-6107-479a-a24a-f4d60ef818d9/volumes"
Jul 17 13:09:44.672950 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:09:44.672918    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:10:09.302452 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:09.302420    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw"]
Jul 17 13:10:09.305102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:09.303022    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86d00aeb-849b-4dcb-9309-caa63933b365" containerName="revoke-keys"
Jul 17 13:10:09.305102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:09.303037    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="86d00aeb-849b-4dcb-9309-caa63933b365" containerName="revoke-keys"
Jul 17 13:10:09.305102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:09.303044    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c43ead2d-6107-479a-a24a-f4d60ef818d9" containerName="revoke-keys"
Jul 17 13:10:09.305102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:09.303050    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c43ead2d-6107-479a-a24a-f4d60ef818d9" containerName="revoke-keys"
Jul 17 13:10:09.305102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:09.303056    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c43ead2d-6107-479a-a24a-f4d60ef818d9" containerName="revoke-keys"
Jul 17 13:10:09.305102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:09.303062    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c43ead2d-6107-479a-a24a-f4d60ef818d9" containerName="revoke-keys"
Jul 17 13:10:09.305102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:09.303069    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c43ead2d-6107-479a-a24a-f4d60ef818d9" containerName="revoke-keys"
Jul 17 13:10:09.305102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:09.303074    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c43ead2d-6107-479a-a24a-f4d60ef818d9" containerName="revoke-keys"
Jul 17 13:10:09.305102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:09.303154    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c43ead2d-6107-479a-a24a-f4d60ef818d9" containerName="revoke-keys"
Jul 17 13:10:09.305102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:09.303163    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c43ead2d-6107-479a-a24a-f4d60ef818d9" containerName="revoke-keys"
Jul 17 13:10:09.305793 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:09.305778    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw"
Jul 17 13:10:09.308811 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:09.308790    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:10:09.316396 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:09.316364    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw"]
Jul 17 13:10:09.356933 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:09.356911    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3736db8b-e144-4180-89a6-5dd6ea53c2ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw\" (UID: \"3736db8b-e144-4180-89a6-5dd6ea53c2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw"
Jul 17 13:10:09.457802 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:09.457776    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3736db8b-e144-4180-89a6-5dd6ea53c2ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw\" (UID: \"3736db8b-e144-4180-89a6-5dd6ea53c2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw"
Jul 17 13:10:09.458385 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:09.458368    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3736db8b-e144-4180-89a6-5dd6ea53c2ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw\" (UID: \"3736db8b-e144-4180-89a6-5dd6ea53c2ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw"
Jul 17 13:10:09.617996 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:09.617931    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw"
Jul 17 13:10:09.756450 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:09.756424    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw"]
Jul 17 13:10:09.757583 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:10:09.757555    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3736db8b_e144_4180_89a6_5dd6ea53c2ba.slice/crio-4914db0ca8cc0bba904af46ce1cead16a6b82b091ac6c03902c8769aedaee32d WatchSource:0}: Error finding container 4914db0ca8cc0bba904af46ce1cead16a6b82b091ac6c03902c8769aedaee32d: Status 404 returned error can't find the container with id 4914db0ca8cc0bba904af46ce1cead16a6b82b091ac6c03902c8769aedaee32d
Jul 17 13:10:09.810345 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:09.810312    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw" event={"ID":"3736db8b-e144-4180-89a6-5dd6ea53c2ba","Type":"ContainerStarted","Data":"4914db0ca8cc0bba904af46ce1cead16a6b82b091ac6c03902c8769aedaee32d"}
Jul 17 13:10:10.817440 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:10.817405    2565 generic.go:356] "Generic (PLEG): container finished" podID="3736db8b-e144-4180-89a6-5dd6ea53c2ba" containerID="074590c13cd901d2f7161cc9e6347d38a3108d00063604d1e002369249f0f639" exitCode=7
Jul 17 13:10:10.817786 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:10.817452    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw" event={"ID":"3736db8b-e144-4180-89a6-5dd6ea53c2ba","Type":"ContainerDied","Data":"074590c13cd901d2f7161cc9e6347d38a3108d00063604d1e002369249f0f639"}
Jul 17 13:10:10.817786 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:10.817666    2565 scope.go:117] "RemoveContainer" containerID="074590c13cd901d2f7161cc9e6347d38a3108d00063604d1e002369249f0f639"
Jul 17 13:10:11.824388 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:11.824361    2565 generic.go:356] "Generic (PLEG): container finished" podID="3736db8b-e144-4180-89a6-5dd6ea53c2ba" containerID="62eb33b950cb70978b190849c45d5d09b3bec51c387194b19aa4b2d362394691" exitCode=7
Jul 17 13:10:11.824740 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:11.824406    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw" event={"ID":"3736db8b-e144-4180-89a6-5dd6ea53c2ba","Type":"ContainerDied","Data":"62eb33b950cb70978b190849c45d5d09b3bec51c387194b19aa4b2d362394691"}
Jul 17 13:10:11.824740 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:11.824460    2565 scope.go:117] "RemoveContainer" containerID="074590c13cd901d2f7161cc9e6347d38a3108d00063604d1e002369249f0f639"
Jul 17 13:10:11.824740 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:11.824596    2565 scope.go:117] "RemoveContainer" containerID="62eb33b950cb70978b190849c45d5d09b3bec51c387194b19aa4b2d362394691"
Jul 17 13:10:11.824868 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:10:11.824784    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw_odh-ai-gateway-infra(3736db8b-e144-4180-89a6-5dd6ea53c2ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw" podUID="3736db8b-e144-4180-89a6-5dd6ea53c2ba"
Jul 17 13:10:12.831471 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:12.831444    2565 scope.go:117] "RemoveContainer" containerID="62eb33b950cb70978b190849c45d5d09b3bec51c387194b19aa4b2d362394691"
Jul 17 13:10:12.831834 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:10:12.831616    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw_odh-ai-gateway-infra(3736db8b-e144-4180-89a6-5dd6ea53c2ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw" podUID="3736db8b-e144-4180-89a6-5dd6ea53c2ba"
Jul 17 13:10:26.400671 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:26.400642    2565 scope.go:117] "RemoveContainer" containerID="62eb33b950cb70978b190849c45d5d09b3bec51c387194b19aa4b2d362394691"
Jul 17 13:10:26.911735 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:26.911667    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw" event={"ID":"3736db8b-e144-4180-89a6-5dd6ea53c2ba","Type":"ContainerStarted","Data":"0747aa933afe79f1570a7740683c10c3a30e1c9054bb076b6ea4e348a9d5ae8e"}
Jul 17 13:10:26.931901 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:26.931849    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw" podStartSLOduration=17.931830764 podStartE2EDuration="17.931830764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:10:26.930326331 +0000 UTC m=+2871.061711006" watchObservedRunningTime="2026-07-17 13:10:26.931830764 +0000 UTC m=+2871.063215426"
Jul 17 13:10:27.918758 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:27.918726    2565 generic.go:356] "Generic (PLEG): container finished" podID="3736db8b-e144-4180-89a6-5dd6ea53c2ba" containerID="0747aa933afe79f1570a7740683c10c3a30e1c9054bb076b6ea4e348a9d5ae8e" exitCode=7
Jul 17 13:10:27.919207 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:27.918802    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw" event={"ID":"3736db8b-e144-4180-89a6-5dd6ea53c2ba","Type":"ContainerDied","Data":"0747aa933afe79f1570a7740683c10c3a30e1c9054bb076b6ea4e348a9d5ae8e"}
Jul 17 13:10:27.919207 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:27.918874    2565 scope.go:117] "RemoveContainer" containerID="62eb33b950cb70978b190849c45d5d09b3bec51c387194b19aa4b2d362394691"
Jul 17 13:10:27.919207 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:27.919170    2565 scope.go:117] "RemoveContainer" containerID="0747aa933afe79f1570a7740683c10c3a30e1c9054bb076b6ea4e348a9d5ae8e"
Jul 17 13:10:27.919376 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:10:27.919339    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw_odh-ai-gateway-infra(3736db8b-e144-4180-89a6-5dd6ea53c2ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw" podUID="3736db8b-e144-4180-89a6-5dd6ea53c2ba"
Jul 17 13:10:27.947630 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:27.947607    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw"]
Jul 17 13:10:29.058782 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:29.058758    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw"
Jul 17 13:10:29.214893 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:29.214820    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3736db8b-e144-4180-89a6-5dd6ea53c2ba-maas-api-service-ca\") pod \"3736db8b-e144-4180-89a6-5dd6ea53c2ba\" (UID: \"3736db8b-e144-4180-89a6-5dd6ea53c2ba\") "
Jul 17 13:10:29.215170 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:29.215149    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3736db8b-e144-4180-89a6-5dd6ea53c2ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3736db8b-e144-4180-89a6-5dd6ea53c2ba" (UID: "3736db8b-e144-4180-89a6-5dd6ea53c2ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:10:29.316285 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:29.316251    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3736db8b-e144-4180-89a6-5dd6ea53c2ba-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:10:29.932023 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:29.931999    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw"
Jul 17 13:10:29.932023 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:29.932018    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw" event={"ID":"3736db8b-e144-4180-89a6-5dd6ea53c2ba","Type":"ContainerDied","Data":"4914db0ca8cc0bba904af46ce1cead16a6b82b091ac6c03902c8769aedaee32d"}
Jul 17 13:10:29.932224 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:29.932076    2565 scope.go:117] "RemoveContainer" containerID="0747aa933afe79f1570a7740683c10c3a30e1c9054bb076b6ea4e348a9d5ae8e"
Jul 17 13:10:29.957271 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:29.957249    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw"]
Jul 17 13:10:29.969660 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:29.969625    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-b2ccw"]
Jul 17 13:10:30.402606 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:10:30.402578    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3736db8b-e144-4180-89a6-5dd6ea53c2ba" path="/var/lib/kubelet/pods/3736db8b-e144-4180-89a6-5dd6ea53c2ba/volumes"
Jul 17 13:11:06.341917 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:06.341874    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:11:07.516292 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:07.516256    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:11:09.386931 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:09.386891    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv"]
Jul 17 13:11:09.387529 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:09.387508    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3736db8b-e144-4180-89a6-5dd6ea53c2ba" containerName="revoke-keys"
Jul 17 13:11:09.387620 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:09.387532    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3736db8b-e144-4180-89a6-5dd6ea53c2ba" containerName="revoke-keys"
Jul 17 13:11:09.387620 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:09.387584    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3736db8b-e144-4180-89a6-5dd6ea53c2ba" containerName="revoke-keys"
Jul 17 13:11:09.387620 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:09.387593    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3736db8b-e144-4180-89a6-5dd6ea53c2ba" containerName="revoke-keys"
Jul 17 13:11:09.387620 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:09.387606    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3736db8b-e144-4180-89a6-5dd6ea53c2ba" containerName="revoke-keys"
Jul 17 13:11:09.387620 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:09.387614    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3736db8b-e144-4180-89a6-5dd6ea53c2ba" containerName="revoke-keys"
Jul 17 13:11:09.387867 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:09.387713    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3736db8b-e144-4180-89a6-5dd6ea53c2ba" containerName="revoke-keys"
Jul 17 13:11:09.387867 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:09.387727    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c43ead2d-6107-479a-a24a-f4d60ef818d9" containerName="revoke-keys"
Jul 17 13:11:09.387867 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:09.387737    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3736db8b-e144-4180-89a6-5dd6ea53c2ba" containerName="revoke-keys"
Jul 17 13:11:09.391052 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:09.391030    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv"
Jul 17 13:11:09.393862 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:09.393840    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:11:09.411337 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:09.411310    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv"]
Jul 17 13:11:09.453394 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:09.453363    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b108f52b-7633-4f46-b47c-e6f2dad9cd63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv\" (UID: \"b108f52b-7633-4f46-b47c-e6f2dad9cd63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv"
Jul 17 13:11:09.554537 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:09.554502    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b108f52b-7633-4f46-b47c-e6f2dad9cd63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv\" (UID: \"b108f52b-7633-4f46-b47c-e6f2dad9cd63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv"
Jul 17 13:11:09.555194 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:09.555169    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b108f52b-7633-4f46-b47c-e6f2dad9cd63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv\" (UID: \"b108f52b-7633-4f46-b47c-e6f2dad9cd63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv"
Jul 17 13:11:09.710186 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:09.710079    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv"
Jul 17 13:11:09.853347 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:09.853322    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv"]
Jul 17 13:11:09.854781 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:11:09.854752    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb108f52b_7633_4f46_b47c_e6f2dad9cd63.slice/crio-a36dc1d5485163d83deb4cb4e43a1b7d0df3133a045adc433f1bfc0a37b07842 WatchSource:0}: Error finding container a36dc1d5485163d83deb4cb4e43a1b7d0df3133a045adc433f1bfc0a37b07842: Status 404 returned error can't find the container with id a36dc1d5485163d83deb4cb4e43a1b7d0df3133a045adc433f1bfc0a37b07842
Jul 17 13:11:10.028137 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:10.028113    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:11:10.130013 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:10.129958    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:11:10.162641 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:10.162615    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv" event={"ID":"b108f52b-7633-4f46-b47c-e6f2dad9cd63","Type":"ContainerStarted","Data":"a9325dc670921c981fd6bd9aec45f37f5aa5b8eb316f2090ee064167709b16aa"}
Jul 17 13:11:10.162792 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:10.162668    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv" event={"ID":"b108f52b-7633-4f46-b47c-e6f2dad9cd63","Type":"ContainerStarted","Data":"a36dc1d5485163d83deb4cb4e43a1b7d0df3133a045adc433f1bfc0a37b07842"}
Jul 17 13:11:10.181533 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:10.181488    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv" podStartSLOduration=1.181473161 podStartE2EDuration="1.181473161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:11:10.180426559 +0000 UTC m=+2914.311811220" watchObservedRunningTime="2026-07-17 13:11:10.181473161 +0000 UTC m=+2914.312857822"
Jul 17 13:11:11.172382 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:11.172349    2565 generic.go:356] "Generic (PLEG): container finished" podID="b108f52b-7633-4f46-b47c-e6f2dad9cd63" containerID="a9325dc670921c981fd6bd9aec45f37f5aa5b8eb316f2090ee064167709b16aa" exitCode=7
Jul 17 13:11:11.172850 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:11.172433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv" event={"ID":"b108f52b-7633-4f46-b47c-e6f2dad9cd63","Type":"ContainerDied","Data":"a9325dc670921c981fd6bd9aec45f37f5aa5b8eb316f2090ee064167709b16aa"}
Jul 17 13:11:11.172850 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:11.172640    2565 scope.go:117] "RemoveContainer" containerID="a9325dc670921c981fd6bd9aec45f37f5aa5b8eb316f2090ee064167709b16aa"
Jul 17 13:11:12.180130 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:12.180038    2565 generic.go:356] "Generic (PLEG): container finished" podID="b108f52b-7633-4f46-b47c-e6f2dad9cd63" containerID="1b7be497c38a6ae77d5b5e2f063222c3d7fc782e4a6c7dbe75cfcd510518b8fb" exitCode=7
Jul 17 13:11:12.180507 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:12.180147    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv" event={"ID":"b108f52b-7633-4f46-b47c-e6f2dad9cd63","Type":"ContainerDied","Data":"1b7be497c38a6ae77d5b5e2f063222c3d7fc782e4a6c7dbe75cfcd510518b8fb"}
Jul 17 13:11:12.180507 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:12.180192    2565 scope.go:117] "RemoveContainer" containerID="a9325dc670921c981fd6bd9aec45f37f5aa5b8eb316f2090ee064167709b16aa"
Jul 17 13:11:12.180507 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:12.180436    2565 scope.go:117] "RemoveContainer" containerID="1b7be497c38a6ae77d5b5e2f063222c3d7fc782e4a6c7dbe75cfcd510518b8fb"
Jul 17 13:11:12.180665 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:11:12.180645    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv_odh-ai-gateway-infra(b108f52b-7633-4f46-b47c-e6f2dad9cd63)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv" podUID="b108f52b-7633-4f46-b47c-e6f2dad9cd63"
Jul 17 13:11:13.187809 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:13.187777    2565 scope.go:117] "RemoveContainer" containerID="1b7be497c38a6ae77d5b5e2f063222c3d7fc782e4a6c7dbe75cfcd510518b8fb"
Jul 17 13:11:13.188192 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:11:13.187978    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv_odh-ai-gateway-infra(b108f52b-7633-4f46-b47c-e6f2dad9cd63)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv" podUID="b108f52b-7633-4f46-b47c-e6f2dad9cd63"
Jul 17 13:11:28.397072 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:28.397043    2565 scope.go:117] "RemoveContainer" containerID="1b7be497c38a6ae77d5b5e2f063222c3d7fc782e4a6c7dbe75cfcd510518b8fb"
Jul 17 13:11:29.281425 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:29.281334    2565 generic.go:356] "Generic (PLEG): container finished" podID="b108f52b-7633-4f46-b47c-e6f2dad9cd63" containerID="6ada03323328de935c52d7b61c554ca89af03631673e77602851564a79ccbd18" exitCode=7
Jul 17 13:11:29.281565 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:29.281408    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv" event={"ID":"b108f52b-7633-4f46-b47c-e6f2dad9cd63","Type":"ContainerDied","Data":"6ada03323328de935c52d7b61c554ca89af03631673e77602851564a79ccbd18"}
Jul 17 13:11:29.281565 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:29.281502    2565 scope.go:117] "RemoveContainer" containerID="1b7be497c38a6ae77d5b5e2f063222c3d7fc782e4a6c7dbe75cfcd510518b8fb"
Jul 17 13:11:29.281864 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:29.281848    2565 scope.go:117] "RemoveContainer" containerID="6ada03323328de935c52d7b61c554ca89af03631673e77602851564a79ccbd18"
Jul 17 13:11:29.282074 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:11:29.282051    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv_odh-ai-gateway-infra(b108f52b-7633-4f46-b47c-e6f2dad9cd63)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv" podUID="b108f52b-7633-4f46-b47c-e6f2dad9cd63"
Jul 17 13:11:30.309638 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:30.309605    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv"]
Jul 17 13:11:30.455866 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:30.455837    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv"
Jul 17 13:11:30.544676 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:30.544643    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b108f52b-7633-4f46-b47c-e6f2dad9cd63-maas-api-service-ca\") pod \"b108f52b-7633-4f46-b47c-e6f2dad9cd63\" (UID: \"b108f52b-7633-4f46-b47c-e6f2dad9cd63\") "
Jul 17 13:11:30.545054 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:30.545031    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b108f52b-7633-4f46-b47c-e6f2dad9cd63-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b108f52b-7633-4f46-b47c-e6f2dad9cd63" (UID: "b108f52b-7633-4f46-b47c-e6f2dad9cd63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:11:30.645660 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:30.645581    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b108f52b-7633-4f46-b47c-e6f2dad9cd63-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:11:31.295534 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:31.295505    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv"
Jul 17 13:11:31.295534 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:31.295532    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv" event={"ID":"b108f52b-7633-4f46-b47c-e6f2dad9cd63","Type":"ContainerDied","Data":"a36dc1d5485163d83deb4cb4e43a1b7d0df3133a045adc433f1bfc0a37b07842"}
Jul 17 13:11:31.295785 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:31.295584    2565 scope.go:117] "RemoveContainer" containerID="6ada03323328de935c52d7b61c554ca89af03631673e77602851564a79ccbd18"
Jul 17 13:11:31.317424 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:31.317388    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv"]
Jul 17 13:11:31.324111 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:31.324086    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-rj2pv"]
Jul 17 13:11:32.403499 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:11:32.403468    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b108f52b-7633-4f46-b47c-e6f2dad9cd63" path="/var/lib/kubelet/pods/b108f52b-7633-4f46-b47c-e6f2dad9cd63/volumes"
Jul 17 13:12:09.458051 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:09.457938    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq"]
Jul 17 13:12:09.458608 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:09.458441    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b108f52b-7633-4f46-b47c-e6f2dad9cd63" containerName="revoke-keys"
Jul 17 13:12:09.458608 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:09.458458    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b108f52b-7633-4f46-b47c-e6f2dad9cd63" containerName="revoke-keys"
Jul 17 13:12:09.458608 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:09.458470    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b108f52b-7633-4f46-b47c-e6f2dad9cd63" containerName="revoke-keys"
Jul 17 13:12:09.458608 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:09.458476    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b108f52b-7633-4f46-b47c-e6f2dad9cd63" containerName="revoke-keys"
Jul 17 13:12:09.458608 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:09.458486    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b108f52b-7633-4f46-b47c-e6f2dad9cd63" containerName="revoke-keys"
Jul 17 13:12:09.458608 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:09.458536    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b108f52b-7633-4f46-b47c-e6f2dad9cd63" containerName="revoke-keys"
Jul 17 13:12:09.458930 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:09.458621    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3736db8b-e144-4180-89a6-5dd6ea53c2ba" containerName="revoke-keys"
Jul 17 13:12:09.458930 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:09.458633    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b108f52b-7633-4f46-b47c-e6f2dad9cd63" containerName="revoke-keys"
Jul 17 13:12:09.458930 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:09.458641    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b108f52b-7633-4f46-b47c-e6f2dad9cd63" containerName="revoke-keys"
Jul 17 13:12:09.461388 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:09.461366    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq"
Jul 17 13:12:09.465317 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:09.465297    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:12:09.474081 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:09.474056    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq"]
Jul 17 13:12:09.575355 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:09.575323    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/032a164c-7875-49e4-852f-6aca1a1fd2a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq\" (UID: \"032a164c-7875-49e4-852f-6aca1a1fd2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq"
Jul 17 13:12:09.676698 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:09.676660    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/032a164c-7875-49e4-852f-6aca1a1fd2a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq\" (UID: \"032a164c-7875-49e4-852f-6aca1a1fd2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq"
Jul 17 13:12:09.677366 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:09.677346    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/032a164c-7875-49e4-852f-6aca1a1fd2a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq\" (UID: \"032a164c-7875-49e4-852f-6aca1a1fd2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq"
Jul 17 13:12:09.772863 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:09.772788    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq"
Jul 17 13:12:09.920012 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:09.919109    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq"]
Jul 17 13:12:10.525810 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:10.525775    2565 generic.go:356] "Generic (PLEG): container finished" podID="032a164c-7875-49e4-852f-6aca1a1fd2a5" containerID="fb2c11fbe9d5c55b9770b7de29ffc4886b903c51e38c392c2018766321057337" exitCode=7
Jul 17 13:12:10.526161 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:10.525857    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq" event={"ID":"032a164c-7875-49e4-852f-6aca1a1fd2a5","Type":"ContainerDied","Data":"fb2c11fbe9d5c55b9770b7de29ffc4886b903c51e38c392c2018766321057337"}
Jul 17 13:12:10.526161 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:10.525906    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq" event={"ID":"032a164c-7875-49e4-852f-6aca1a1fd2a5","Type":"ContainerStarted","Data":"09d5f1c1dacf20005b57e2a9137e5cec28ed92170c585ece171978c809bf79c5"}
Jul 17 13:12:10.526161 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:10.526038    2565 scope.go:117] "RemoveContainer" containerID="fb2c11fbe9d5c55b9770b7de29ffc4886b903c51e38c392c2018766321057337"
Jul 17 13:12:11.533943 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:11.533909    2565 generic.go:356] "Generic (PLEG): container finished" podID="032a164c-7875-49e4-852f-6aca1a1fd2a5" containerID="3c51994d1684595b786258afd4325e2470a0aab64e427fce982d9f644b9b5cc2" exitCode=7
Jul 17 13:12:11.534335 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:11.534009    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq" event={"ID":"032a164c-7875-49e4-852f-6aca1a1fd2a5","Type":"ContainerDied","Data":"3c51994d1684595b786258afd4325e2470a0aab64e427fce982d9f644b9b5cc2"}
Jul 17 13:12:11.534335 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:11.534076    2565 scope.go:117] "RemoveContainer" containerID="fb2c11fbe9d5c55b9770b7de29ffc4886b903c51e38c392c2018766321057337"
Jul 17 13:12:11.534335 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:11.534209    2565 scope.go:117] "RemoveContainer" containerID="3c51994d1684595b786258afd4325e2470a0aab64e427fce982d9f644b9b5cc2"
Jul 17 13:12:11.534469 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:12:11.534425    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq_odh-ai-gateway-infra(032a164c-7875-49e4-852f-6aca1a1fd2a5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq" podUID="032a164c-7875-49e4-852f-6aca1a1fd2a5"
Jul 17 13:12:12.541721 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:12.541694    2565 scope.go:117] "RemoveContainer" containerID="3c51994d1684595b786258afd4325e2470a0aab64e427fce982d9f644b9b5cc2"
Jul 17 13:12:12.542122 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:12:12.541887    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq_odh-ai-gateway-infra(032a164c-7875-49e4-852f-6aca1a1fd2a5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq" podUID="032a164c-7875-49e4-852f-6aca1a1fd2a5"
Jul 17 13:12:27.397399 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:27.397363    2565 scope.go:117] "RemoveContainer" containerID="3c51994d1684595b786258afd4325e2470a0aab64e427fce982d9f644b9b5cc2"
Jul 17 13:12:28.633763 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:28.633732    2565 generic.go:356] "Generic (PLEG): container finished" podID="032a164c-7875-49e4-852f-6aca1a1fd2a5" containerID="82a30929d4f25d410df07325f33ba7eb9fcb4afffe06af3d5b9a1fde9c697ff0" exitCode=7
Jul 17 13:12:28.634205 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:28.633805    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq" event={"ID":"032a164c-7875-49e4-852f-6aca1a1fd2a5","Type":"ContainerDied","Data":"82a30929d4f25d410df07325f33ba7eb9fcb4afffe06af3d5b9a1fde9c697ff0"}
Jul 17 13:12:28.634205 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:28.633881    2565 scope.go:117] "RemoveContainer" containerID="3c51994d1684595b786258afd4325e2470a0aab64e427fce982d9f644b9b5cc2"
Jul 17 13:12:28.634205 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:28.634202    2565 scope.go:117] "RemoveContainer" containerID="82a30929d4f25d410df07325f33ba7eb9fcb4afffe06af3d5b9a1fde9c697ff0"
Jul 17 13:12:28.634387 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:12:28.634368    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq_odh-ai-gateway-infra(032a164c-7875-49e4-852f-6aca1a1fd2a5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq" podUID="032a164c-7875-49e4-852f-6aca1a1fd2a5"
Jul 17 13:12:29.663404 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:29.663370    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq"]
Jul 17 13:12:29.804728 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:29.804695    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq"
Jul 17 13:12:29.950319 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:29.950246    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/032a164c-7875-49e4-852f-6aca1a1fd2a5-maas-api-service-ca\") pod \"032a164c-7875-49e4-852f-6aca1a1fd2a5\" (UID: \"032a164c-7875-49e4-852f-6aca1a1fd2a5\") "
Jul 17 13:12:29.950580 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:29.950557    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/032a164c-7875-49e4-852f-6aca1a1fd2a5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "032a164c-7875-49e4-852f-6aca1a1fd2a5" (UID: "032a164c-7875-49e4-852f-6aca1a1fd2a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:12:30.050874 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:30.050848    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/032a164c-7875-49e4-852f-6aca1a1fd2a5-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:12:30.649401 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:30.649374    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq"
Jul 17 13:12:30.649589 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:30.649420    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq" event={"ID":"032a164c-7875-49e4-852f-6aca1a1fd2a5","Type":"ContainerDied","Data":"09d5f1c1dacf20005b57e2a9137e5cec28ed92170c585ece171978c809bf79c5"}
Jul 17 13:12:30.649589 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:30.649478    2565 scope.go:117] "RemoveContainer" containerID="82a30929d4f25d410df07325f33ba7eb9fcb4afffe06af3d5b9a1fde9c697ff0"
Jul 17 13:12:30.671109 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:30.671058    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq"]
Jul 17 13:12:30.685990 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:30.685964    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-bt4pq"]
Jul 17 13:12:32.403411 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:32.403381    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="032a164c-7875-49e4-852f-6aca1a1fd2a5" path="/var/lib/kubelet/pods/032a164c-7875-49e4-852f-6aca1a1fd2a5/volumes"
Jul 17 13:12:36.645203 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:36.645174    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 13:12:36.651954 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:36.651934    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 13:12:36.657249 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:36.657227    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 13:12:36.662110 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:36.662093    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 13:12:50.644478 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:50.644448    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz"]
Jul 17 13:12:50.644863 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:50.644850    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="032a164c-7875-49e4-852f-6aca1a1fd2a5" containerName="revoke-keys"
Jul 17 13:12:50.644907 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:50.644864    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="032a164c-7875-49e4-852f-6aca1a1fd2a5" containerName="revoke-keys"
Jul 17 13:12:50.644907 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:50.644877    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="032a164c-7875-49e4-852f-6aca1a1fd2a5" containerName="revoke-keys"
Jul 17 13:12:50.644907 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:50.644883    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="032a164c-7875-49e4-852f-6aca1a1fd2a5" containerName="revoke-keys"
Jul 17 13:12:50.645023 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:50.644972    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b108f52b-7633-4f46-b47c-e6f2dad9cd63" containerName="revoke-keys"
Jul 17 13:12:50.645023 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:50.644996    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="032a164c-7875-49e4-852f-6aca1a1fd2a5" containerName="revoke-keys"
Jul 17 13:12:50.645023 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:50.645005    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="032a164c-7875-49e4-852f-6aca1a1fd2a5" containerName="revoke-keys"
Jul 17 13:12:50.645023 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:50.645010    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="032a164c-7875-49e4-852f-6aca1a1fd2a5" containerName="revoke-keys"
Jul 17 13:12:50.647900 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:50.647884    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz"
Jul 17 13:12:50.651445 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:50.651427    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:12:50.659439 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:50.659414    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz"]
Jul 17 13:12:50.814027 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:50.813976    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-649bz\" (UID: \"a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz"
Jul 17 13:12:50.914629 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:50.914568    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-649bz\" (UID: \"a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz"
Jul 17 13:12:50.915154 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:50.915136    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-649bz\" (UID: \"a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz"
Jul 17 13:12:50.958675 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:50.958653    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz"
Jul 17 13:12:51.097047 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:51.097018    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz"]
Jul 17 13:12:51.098273 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:12:51.098241    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2830ab2_44a6_4067_b2a6_7c4f1baf2c4c.slice/crio-e0555532878259ed20e333ad9b7c07f8c913fac5989a254a1c50383ff06450f7 WatchSource:0}: Error finding container e0555532878259ed20e333ad9b7c07f8c913fac5989a254a1c50383ff06450f7: Status 404 returned error can't find the container with id e0555532878259ed20e333ad9b7c07f8c913fac5989a254a1c50383ff06450f7
Jul 17 13:12:51.774661 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:51.774589    2565 generic.go:356] "Generic (PLEG): container finished" podID="a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c" containerID="00ab4f8fd3c5ca067d9ab41e1198da8127883fa28ad3187ce8a114c4b2916672" exitCode=6
Jul 17 13:12:51.775076 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:51.774798    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz" event={"ID":"a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c","Type":"ContainerDied","Data":"00ab4f8fd3c5ca067d9ab41e1198da8127883fa28ad3187ce8a114c4b2916672"}
Jul 17 13:12:51.775076 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:51.774871    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz" event={"ID":"a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c","Type":"ContainerStarted","Data":"e0555532878259ed20e333ad9b7c07f8c913fac5989a254a1c50383ff06450f7"}
Jul 17 13:12:51.775076 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:51.775055    2565 scope.go:117] "RemoveContainer" containerID="00ab4f8fd3c5ca067d9ab41e1198da8127883fa28ad3187ce8a114c4b2916672"
Jul 17 13:12:52.782525 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:52.782449    2565 generic.go:356] "Generic (PLEG): container finished" podID="a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c" containerID="f46913fd55bbf4c1a39b33f3f44e9e034eae2a840babbb993cbeee54b6847469" exitCode=6
Jul 17 13:12:52.782961 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:52.782528    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz" event={"ID":"a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c","Type":"ContainerDied","Data":"f46913fd55bbf4c1a39b33f3f44e9e034eae2a840babbb993cbeee54b6847469"}
Jul 17 13:12:52.782961 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:52.782585    2565 scope.go:117] "RemoveContainer" containerID="00ab4f8fd3c5ca067d9ab41e1198da8127883fa28ad3187ce8a114c4b2916672"
Jul 17 13:12:52.782961 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:52.782739    2565 scope.go:117] "RemoveContainer" containerID="f46913fd55bbf4c1a39b33f3f44e9e034eae2a840babbb993cbeee54b6847469"
Jul 17 13:12:52.782961 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:12:52.782949    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-649bz_odh-ai-gateway-infra(a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz" podUID="a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c"
Jul 17 13:12:53.791129 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:12:53.791095    2565 scope.go:117] "RemoveContainer" containerID="f46913fd55bbf4c1a39b33f3f44e9e034eae2a840babbb993cbeee54b6847469"
Jul 17 13:12:53.791633 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:12:53.791321    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-649bz_odh-ai-gateway-infra(a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz" podUID="a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c"
Jul 17 13:13:04.397028 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:04.396970    2565 scope.go:117] "RemoveContainer" containerID="f46913fd55bbf4c1a39b33f3f44e9e034eae2a840babbb993cbeee54b6847469"
Jul 17 13:13:04.854894 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:04.854863    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz" event={"ID":"a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c","Type":"ContainerStarted","Data":"0bad0a10d5dafb777e338b2e8a0fe48bb7e418a6260e40c632d7257d6e27446b"}
Jul 17 13:13:04.874124 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:04.874084    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz" podStartSLOduration=14.874069683 podStartE2EDuration="14.874069683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:13:04.872149647 +0000 UTC m=+3029.003534331" watchObservedRunningTime="2026-07-17 13:13:04.874069683 +0000 UTC m=+3029.005454344"
Jul 17 13:13:05.863029 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:05.862976    2565 generic.go:356] "Generic (PLEG): container finished" podID="a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c" containerID="0bad0a10d5dafb777e338b2e8a0fe48bb7e418a6260e40c632d7257d6e27446b" exitCode=6
Jul 17 13:13:05.863507 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:05.863026    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz" event={"ID":"a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c","Type":"ContainerDied","Data":"0bad0a10d5dafb777e338b2e8a0fe48bb7e418a6260e40c632d7257d6e27446b"}
Jul 17 13:13:05.863507 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:05.863088    2565 scope.go:117] "RemoveContainer" containerID="f46913fd55bbf4c1a39b33f3f44e9e034eae2a840babbb993cbeee54b6847469"
Jul 17 13:13:05.863507 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:05.863377    2565 scope.go:117] "RemoveContainer" containerID="0bad0a10d5dafb777e338b2e8a0fe48bb7e418a6260e40c632d7257d6e27446b"
Jul 17 13:13:05.863685 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:13:05.863590    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-649bz_odh-ai-gateway-infra(a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz" podUID="a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c"
Jul 17 13:13:05.882858 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:05.882838    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz"]
Jul 17 13:13:07.012088 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:07.012066    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz"
Jul 17 13:13:07.031778 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:07.031750    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c-maas-api-service-ca\") pod \"a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c\" (UID: \"a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c\") "
Jul 17 13:13:07.032114 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:07.032091    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c" (UID: "a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:13:07.133491 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:07.133432    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:13:07.880787 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:07.880750    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz"
Jul 17 13:13:07.881017 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:07.880786    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz" event={"ID":"a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c","Type":"ContainerDied","Data":"e0555532878259ed20e333ad9b7c07f8c913fac5989a254a1c50383ff06450f7"}
Jul 17 13:13:07.881017 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:07.880863    2565 scope.go:117] "RemoveContainer" containerID="0bad0a10d5dafb777e338b2e8a0fe48bb7e418a6260e40c632d7257d6e27446b"
Jul 17 13:13:07.911167 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:07.911141    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz"]
Jul 17 13:13:07.915164 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:07.915143    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-649bz"]
Jul 17 13:13:08.403191 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:08.403162    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c" path="/var/lib/kubelet/pods/a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c/volumes"
Jul 17 13:13:19.491807 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:19.491776    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:13:19.571949 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:19.571918    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:13:25.863130 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:25.863092    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:13:26.114289 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:26.114212    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:13:32.310076 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:32.310001    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:13:32.419127 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:32.419100    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:13:32.669172 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:32.669142    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:13:33.762947 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:33.762914    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:13:33.893468 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:33.893438    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:13:34.029495 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:34.029403    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:13:34.147432 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:34.147400    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:13:34.591443 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:34.591411    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:13:34.707761 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:34.707727    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:13:36.085548 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:36.085518    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:13:36.144826 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:36.144798    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:13:40.523537 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:13:40.523502    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:14:09.545097 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:09.545067    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w"]
Jul 17 13:14:09.545536 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:09.545474    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c" containerName="revoke-keys"
Jul 17 13:14:09.545536 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:09.545485    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c" containerName="revoke-keys"
Jul 17 13:14:09.545536 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:09.545500    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="032a164c-7875-49e4-852f-6aca1a1fd2a5" containerName="revoke-keys"
Jul 17 13:14:09.545536 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:09.545505    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="032a164c-7875-49e4-852f-6aca1a1fd2a5" containerName="revoke-keys"
Jul 17 13:14:09.545536 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:09.545511    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c" containerName="revoke-keys"
Jul 17 13:14:09.545536 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:09.545517    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c" containerName="revoke-keys"
Jul 17 13:14:09.545536 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:09.545526    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c" containerName="revoke-keys"
Jul 17 13:14:09.545536 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:09.545532    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c" containerName="revoke-keys"
Jul 17 13:14:09.545781 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:09.545595    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c" containerName="revoke-keys"
Jul 17 13:14:09.545781 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:09.545603    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c" containerName="revoke-keys"
Jul 17 13:14:09.545781 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:09.545611    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2830ab2-44a6-4067-b2a6-7c4f1baf2c4c" containerName="revoke-keys"
Jul 17 13:14:09.548274 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:09.548256    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w"
Jul 17 13:14:09.551240 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:09.551222    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:14:09.556231 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:09.556209    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w"]
Jul 17 13:14:09.636679 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:09.636649    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/017fb09c-d35b-40e1-9a6b-400784080a16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w\" (UID: \"017fb09c-d35b-40e1-9a6b-400784080a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w"
Jul 17 13:14:09.737525 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:09.737499    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/017fb09c-d35b-40e1-9a6b-400784080a16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w\" (UID: \"017fb09c-d35b-40e1-9a6b-400784080a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w"
Jul 17 13:14:09.738051 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:09.738035    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/017fb09c-d35b-40e1-9a6b-400784080a16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w\" (UID: \"017fb09c-d35b-40e1-9a6b-400784080a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w"
Jul 17 13:14:09.860097 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:09.860075    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w"
Jul 17 13:14:09.994353 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:09.994324    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w"]
Jul 17 13:14:09.995841 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:14:09.995816    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017fb09c_d35b_40e1_9a6b_400784080a16.slice/crio-3a76b31a77693caa9704fbdab7a1da7e85e949d3a28490bd5a9ae6973401caa5 WatchSource:0}: Error finding container 3a76b31a77693caa9704fbdab7a1da7e85e949d3a28490bd5a9ae6973401caa5: Status 404 returned error can't find the container with id 3a76b31a77693caa9704fbdab7a1da7e85e949d3a28490bd5a9ae6973401caa5
Jul 17 13:14:10.242813 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:10.242783    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w" event={"ID":"017fb09c-d35b-40e1-9a6b-400784080a16","Type":"ContainerStarted","Data":"aa8bcd0c88e0e6b3c4514ee7f51981f4c6f04a641f2ea933603d30b0eb4cf1cb"}
Jul 17 13:14:10.242939 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:10.242846    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w" event={"ID":"017fb09c-d35b-40e1-9a6b-400784080a16","Type":"ContainerStarted","Data":"3a76b31a77693caa9704fbdab7a1da7e85e949d3a28490bd5a9ae6973401caa5"}
Jul 17 13:14:10.263480 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:10.263438    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w" podStartSLOduration=1.2634232029999999 podStartE2EDuration="1.263423203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:14:10.262086081 +0000 UTC m=+3094.393470743" watchObservedRunningTime="2026-07-17 13:14:10.263423203 +0000 UTC m=+3094.394807862"
Jul 17 13:14:11.249961 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:11.249920    2565 generic.go:356] "Generic (PLEG): container finished" podID="017fb09c-d35b-40e1-9a6b-400784080a16" containerID="aa8bcd0c88e0e6b3c4514ee7f51981f4c6f04a641f2ea933603d30b0eb4cf1cb" exitCode=6
Jul 17 13:14:11.250419 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:11.249960    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w" event={"ID":"017fb09c-d35b-40e1-9a6b-400784080a16","Type":"ContainerDied","Data":"aa8bcd0c88e0e6b3c4514ee7f51981f4c6f04a641f2ea933603d30b0eb4cf1cb"}
Jul 17 13:14:11.250419 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:11.250271    2565 scope.go:117] "RemoveContainer" containerID="aa8bcd0c88e0e6b3c4514ee7f51981f4c6f04a641f2ea933603d30b0eb4cf1cb"
Jul 17 13:14:12.257957 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:12.257877    2565 generic.go:356] "Generic (PLEG): container finished" podID="017fb09c-d35b-40e1-9a6b-400784080a16" containerID="08b53b97fce6dcc95e97ab1f86798b2f993a632e24ca6d538f2e7cec38f286c4" exitCode=6
Jul 17 13:14:12.258379 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:12.257963    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w" event={"ID":"017fb09c-d35b-40e1-9a6b-400784080a16","Type":"ContainerDied","Data":"08b53b97fce6dcc95e97ab1f86798b2f993a632e24ca6d538f2e7cec38f286c4"}
Jul 17 13:14:12.258379 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:12.258050    2565 scope.go:117] "RemoveContainer" containerID="aa8bcd0c88e0e6b3c4514ee7f51981f4c6f04a641f2ea933603d30b0eb4cf1cb"
Jul 17 13:14:12.258379 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:12.258140    2565 scope.go:117] "RemoveContainer" containerID="08b53b97fce6dcc95e97ab1f86798b2f993a632e24ca6d538f2e7cec38f286c4"
Jul 17 13:14:12.258379 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:14:12.258342    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w_odh-ai-gateway-infra(017fb09c-d35b-40e1-9a6b-400784080a16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w" podUID="017fb09c-d35b-40e1-9a6b-400784080a16"
Jul 17 13:14:13.267126 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:13.267100    2565 scope.go:117] "RemoveContainer" containerID="08b53b97fce6dcc95e97ab1f86798b2f993a632e24ca6d538f2e7cec38f286c4"
Jul 17 13:14:13.267501 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:14:13.267256    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w_odh-ai-gateway-infra(017fb09c-d35b-40e1-9a6b-400784080a16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w" podUID="017fb09c-d35b-40e1-9a6b-400784080a16"
Jul 17 13:14:27.396332 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:27.396305    2565 scope.go:117] "RemoveContainer" containerID="08b53b97fce6dcc95e97ab1f86798b2f993a632e24ca6d538f2e7cec38f286c4"
Jul 17 13:14:28.351640 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:28.351612    2565 generic.go:356] "Generic (PLEG): container finished" podID="017fb09c-d35b-40e1-9a6b-400784080a16" containerID="4c2e0270876b13364f716f068dd562d6f23e9160b4c600162cbad70106dee73b" exitCode=6
Jul 17 13:14:28.351762 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:28.351686    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w" event={"ID":"017fb09c-d35b-40e1-9a6b-400784080a16","Type":"ContainerDied","Data":"4c2e0270876b13364f716f068dd562d6f23e9160b4c600162cbad70106dee73b"}
Jul 17 13:14:28.351762 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:28.351741    2565 scope.go:117] "RemoveContainer" containerID="08b53b97fce6dcc95e97ab1f86798b2f993a632e24ca6d538f2e7cec38f286c4"
Jul 17 13:14:28.352000 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:28.351974    2565 scope.go:117] "RemoveContainer" containerID="4c2e0270876b13364f716f068dd562d6f23e9160b4c600162cbad70106dee73b"
Jul 17 13:14:28.352218 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:14:28.352196    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w_odh-ai-gateway-infra(017fb09c-d35b-40e1-9a6b-400784080a16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w" podUID="017fb09c-d35b-40e1-9a6b-400784080a16"
Jul 17 13:14:29.396515 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:29.396483    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w"]
Jul 17 13:14:29.537972 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:29.537948    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w"
Jul 17 13:14:29.587896 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:29.587866    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/017fb09c-d35b-40e1-9a6b-400784080a16-maas-api-service-ca\") pod \"017fb09c-d35b-40e1-9a6b-400784080a16\" (UID: \"017fb09c-d35b-40e1-9a6b-400784080a16\") "
Jul 17 13:14:29.588265 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:29.588238    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/017fb09c-d35b-40e1-9a6b-400784080a16-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "017fb09c-d35b-40e1-9a6b-400784080a16" (UID: "017fb09c-d35b-40e1-9a6b-400784080a16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:14:29.689502 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:29.689414    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/017fb09c-d35b-40e1-9a6b-400784080a16-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:14:30.366154 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:30.366126    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w"
Jul 17 13:14:30.366154 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:30.366142    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w" event={"ID":"017fb09c-d35b-40e1-9a6b-400784080a16","Type":"ContainerDied","Data":"3a76b31a77693caa9704fbdab7a1da7e85e949d3a28490bd5a9ae6973401caa5"}
Jul 17 13:14:30.366372 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:30.366201    2565 scope.go:117] "RemoveContainer" containerID="4c2e0270876b13364f716f068dd562d6f23e9160b4c600162cbad70106dee73b"
Jul 17 13:14:30.394865 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:30.394836    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w"]
Jul 17 13:14:30.403046 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:30.403028    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-f6k6w"]
Jul 17 13:14:32.401966 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:32.401926    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="017fb09c-d35b-40e1-9a6b-400784080a16" path="/var/lib/kubelet/pods/017fb09c-d35b-40e1-9a6b-400784080a16/volumes"
Jul 17 13:14:38.248598 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:38.248568    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8"]
Jul 17 13:14:38.248977 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:38.248965    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="017fb09c-d35b-40e1-9a6b-400784080a16" containerName="revoke-keys"
Jul 17 13:14:38.248977 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:38.248976    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="017fb09c-d35b-40e1-9a6b-400784080a16" containerName="revoke-keys"
Jul 17 13:14:38.249098 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:38.249004    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="017fb09c-d35b-40e1-9a6b-400784080a16" containerName="revoke-keys"
Jul 17 13:14:38.249098 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:38.249010    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="017fb09c-d35b-40e1-9a6b-400784080a16" containerName="revoke-keys"
Jul 17 13:14:38.249098 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:38.249052    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="017fb09c-d35b-40e1-9a6b-400784080a16" containerName="revoke-keys"
Jul 17 13:14:38.249098 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:38.249058    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="017fb09c-d35b-40e1-9a6b-400784080a16" containerName="revoke-keys"
Jul 17 13:14:38.249241 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:38.249120    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="017fb09c-d35b-40e1-9a6b-400784080a16" containerName="revoke-keys"
Jul 17 13:14:38.249241 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:38.249129    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="017fb09c-d35b-40e1-9a6b-400784080a16" containerName="revoke-keys"
Jul 17 13:14:38.253263 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:38.253246    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8"
Jul 17 13:14:38.256680 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:38.256655    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:14:38.264520 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:38.264500    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8"]
Jul 17 13:14:38.354910 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:38.354878    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5643137-282c-4242-8ce7-518ebf5945d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8\" (UID: \"f5643137-282c-4242-8ce7-518ebf5945d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8"
Jul 17 13:14:38.455833 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:38.455805    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5643137-282c-4242-8ce7-518ebf5945d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8\" (UID: \"f5643137-282c-4242-8ce7-518ebf5945d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8"
Jul 17 13:14:38.456445 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:38.456425    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5643137-282c-4242-8ce7-518ebf5945d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8\" (UID: \"f5643137-282c-4242-8ce7-518ebf5945d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8"
Jul 17 13:14:38.564030 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:38.564010    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8"
Jul 17 13:14:38.703235 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:38.703205    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8"]
Jul 17 13:14:38.703963 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:14:38.703935    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5643137_282c_4242_8ce7_518ebf5945d6.slice/crio-da40ec6e556b3d34ef3ac32ccd59865bb08f6bb4353551890efbb8dfab41d5fa WatchSource:0}: Error finding container da40ec6e556b3d34ef3ac32ccd59865bb08f6bb4353551890efbb8dfab41d5fa: Status 404 returned error can't find the container with id da40ec6e556b3d34ef3ac32ccd59865bb08f6bb4353551890efbb8dfab41d5fa
Jul 17 13:14:39.419906 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:39.419876    2565 generic.go:356] "Generic (PLEG): container finished" podID="f5643137-282c-4242-8ce7-518ebf5945d6" containerID="046838b416b1c1d27a7603715f09404113a57ad8841168b8114d55a21852841e" exitCode=7
Jul 17 13:14:39.420221 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:39.419932    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8" event={"ID":"f5643137-282c-4242-8ce7-518ebf5945d6","Type":"ContainerDied","Data":"046838b416b1c1d27a7603715f09404113a57ad8841168b8114d55a21852841e"}
Jul 17 13:14:39.420221 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:39.420014    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8" event={"ID":"f5643137-282c-4242-8ce7-518ebf5945d6","Type":"ContainerStarted","Data":"da40ec6e556b3d34ef3ac32ccd59865bb08f6bb4353551890efbb8dfab41d5fa"}
Jul 17 13:14:39.420221 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:39.420185    2565 scope.go:117] "RemoveContainer" containerID="046838b416b1c1d27a7603715f09404113a57ad8841168b8114d55a21852841e"
Jul 17 13:14:40.427680 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:40.427647    2565 generic.go:356] "Generic (PLEG): container finished" podID="f5643137-282c-4242-8ce7-518ebf5945d6" containerID="b43517b2ce22adb928fea8f64e5d8a33788ed70e01bf50e4430b4fd52298d364" exitCode=7
Jul 17 13:14:40.428071 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:40.427737    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8" event={"ID":"f5643137-282c-4242-8ce7-518ebf5945d6","Type":"ContainerDied","Data":"b43517b2ce22adb928fea8f64e5d8a33788ed70e01bf50e4430b4fd52298d364"}
Jul 17 13:14:40.428071 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:40.427794    2565 scope.go:117] "RemoveContainer" containerID="046838b416b1c1d27a7603715f09404113a57ad8841168b8114d55a21852841e"
Jul 17 13:14:40.428071 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:40.427904    2565 scope.go:117] "RemoveContainer" containerID="b43517b2ce22adb928fea8f64e5d8a33788ed70e01bf50e4430b4fd52298d364"
Jul 17 13:14:40.428262 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:14:40.428090    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8_odh-ai-gateway-infra(f5643137-282c-4242-8ce7-518ebf5945d6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8" podUID="f5643137-282c-4242-8ce7-518ebf5945d6"
Jul 17 13:14:41.437214 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:41.437190    2565 scope.go:117] "RemoveContainer" containerID="b43517b2ce22adb928fea8f64e5d8a33788ed70e01bf50e4430b4fd52298d364"
Jul 17 13:14:41.437549 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:14:41.437378    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8_odh-ai-gateway-infra(f5643137-282c-4242-8ce7-518ebf5945d6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8" podUID="f5643137-282c-4242-8ce7-518ebf5945d6"
Jul 17 13:14:53.396864 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:53.396833    2565 scope.go:117] "RemoveContainer" containerID="b43517b2ce22adb928fea8f64e5d8a33788ed70e01bf50e4430b4fd52298d364"
Jul 17 13:14:54.508031 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:54.507973    2565 generic.go:356] "Generic (PLEG): container finished" podID="f5643137-282c-4242-8ce7-518ebf5945d6" containerID="f31d934f7e1a46822c2d17fe23c240920648f9e8d3189922f9b618387c80ffd7" exitCode=7
Jul 17 13:14:54.508424 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:54.508050    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8" event={"ID":"f5643137-282c-4242-8ce7-518ebf5945d6","Type":"ContainerDied","Data":"f31d934f7e1a46822c2d17fe23c240920648f9e8d3189922f9b618387c80ffd7"}
Jul 17 13:14:54.508424 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:54.508102    2565 scope.go:117] "RemoveContainer" containerID="b43517b2ce22adb928fea8f64e5d8a33788ed70e01bf50e4430b4fd52298d364"
Jul 17 13:14:54.508424 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:54.508294    2565 scope.go:117] "RemoveContainer" containerID="f31d934f7e1a46822c2d17fe23c240920648f9e8d3189922f9b618387c80ffd7"
Jul 17 13:14:54.508567 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:14:54.508470    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8_odh-ai-gateway-infra(f5643137-282c-4242-8ce7-518ebf5945d6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8" podUID="f5643137-282c-4242-8ce7-518ebf5945d6"
Jul 17 13:14:55.542304 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:55.542273    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8"]
Jul 17 13:14:55.675492 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:55.675471    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8"
Jul 17 13:14:55.794109 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:55.794039    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5643137-282c-4242-8ce7-518ebf5945d6-maas-api-service-ca\") pod \"f5643137-282c-4242-8ce7-518ebf5945d6\" (UID: \"f5643137-282c-4242-8ce7-518ebf5945d6\") "
Jul 17 13:14:55.794346 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:55.794326    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5643137-282c-4242-8ce7-518ebf5945d6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5643137-282c-4242-8ce7-518ebf5945d6" (UID: "f5643137-282c-4242-8ce7-518ebf5945d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:14:55.895385 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:55.895359    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5643137-282c-4242-8ce7-518ebf5945d6-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:14:56.522475 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:56.522448    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8"
Jul 17 13:14:56.522475 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:56.522451    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8" event={"ID":"f5643137-282c-4242-8ce7-518ebf5945d6","Type":"ContainerDied","Data":"da40ec6e556b3d34ef3ac32ccd59865bb08f6bb4353551890efbb8dfab41d5fa"}
Jul 17 13:14:56.522673 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:56.522503    2565 scope.go:117] "RemoveContainer" containerID="f31d934f7e1a46822c2d17fe23c240920648f9e8d3189922f9b618387c80ffd7"
Jul 17 13:14:56.549717 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:56.549682    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8"]
Jul 17 13:14:56.550132 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:56.549742    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hhxp8"]
Jul 17 13:14:58.403978 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:14:58.403943    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5643137-282c-4242-8ce7-518ebf5945d6" path="/var/lib/kubelet/pods/f5643137-282c-4242-8ce7-518ebf5945d6/volumes"
Jul 17 13:15:09.619514 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:09.619481    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs"]
Jul 17 13:15:09.621282 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:09.619906    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5643137-282c-4242-8ce7-518ebf5945d6" containerName="revoke-keys"
Jul 17 13:15:09.621282 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:09.619921    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5643137-282c-4242-8ce7-518ebf5945d6" containerName="revoke-keys"
Jul 17 13:15:09.621282 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:09.619946    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5643137-282c-4242-8ce7-518ebf5945d6" containerName="revoke-keys"
Jul 17 13:15:09.621282 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:09.619952    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5643137-282c-4242-8ce7-518ebf5945d6" containerName="revoke-keys"
Jul 17 13:15:09.621282 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:09.620033    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="017fb09c-d35b-40e1-9a6b-400784080a16" containerName="revoke-keys"
Jul 17 13:15:09.621282 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:09.620044    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5643137-282c-4242-8ce7-518ebf5945d6" containerName="revoke-keys"
Jul 17 13:15:09.621282 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:09.620052    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5643137-282c-4242-8ce7-518ebf5945d6" containerName="revoke-keys"
Jul 17 13:15:09.621282 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:09.620057    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5643137-282c-4242-8ce7-518ebf5945d6" containerName="revoke-keys"
Jul 17 13:15:09.622523 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:09.622508    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs"
Jul 17 13:15:09.625749 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:09.625733    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:15:09.641203 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:09.639706    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs"]
Jul 17 13:15:09.800701 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:09.800668    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29cd9c58-050a-4766-b1e3-1ce15cdfefe4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs\" (UID: \"29cd9c58-050a-4766-b1e3-1ce15cdfefe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs"
Jul 17 13:15:09.901505 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:09.901448    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29cd9c58-050a-4766-b1e3-1ce15cdfefe4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs\" (UID: \"29cd9c58-050a-4766-b1e3-1ce15cdfefe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs"
Jul 17 13:15:09.902090 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:09.902069    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29cd9c58-050a-4766-b1e3-1ce15cdfefe4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs\" (UID: \"29cd9c58-050a-4766-b1e3-1ce15cdfefe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs"
Jul 17 13:15:09.932710 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:09.932691    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs"
Jul 17 13:15:10.275338 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:10.275315    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs"]
Jul 17 13:15:10.276406 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:15:10.276381    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29cd9c58_050a_4766_b1e3_1ce15cdfefe4.slice/crio-9143b393b1c4b5e1f5c480519563b7ccf429abd81e6072fb359cbe45026f44ec WatchSource:0}: Error finding container 9143b393b1c4b5e1f5c480519563b7ccf429abd81e6072fb359cbe45026f44ec: Status 404 returned error can't find the container with id 9143b393b1c4b5e1f5c480519563b7ccf429abd81e6072fb359cbe45026f44ec
Jul 17 13:15:10.603389 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:10.603363    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs" event={"ID":"29cd9c58-050a-4766-b1e3-1ce15cdfefe4","Type":"ContainerStarted","Data":"94ce740235742e994268234c0b7c6ac490ef8057420dfc8498872c9ccf7067c6"}
Jul 17 13:15:10.603494 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:10.603409    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs" event={"ID":"29cd9c58-050a-4766-b1e3-1ce15cdfefe4","Type":"ContainerStarted","Data":"9143b393b1c4b5e1f5c480519563b7ccf429abd81e6072fb359cbe45026f44ec"}
Jul 17 13:15:10.630429 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:10.630389    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs" podStartSLOduration=1.630375709 podStartE2EDuration="1.630375709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:15:10.627006494 +0000 UTC m=+3154.758391147" watchObservedRunningTime="2026-07-17 13:15:10.630375709 +0000 UTC m=+3154.761760397"
Jul 17 13:15:11.610749 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:11.610723    2565 generic.go:356] "Generic (PLEG): container finished" podID="29cd9c58-050a-4766-b1e3-1ce15cdfefe4" containerID="94ce740235742e994268234c0b7c6ac490ef8057420dfc8498872c9ccf7067c6" exitCode=6
Jul 17 13:15:11.610908 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:11.610817    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs" event={"ID":"29cd9c58-050a-4766-b1e3-1ce15cdfefe4","Type":"ContainerDied","Data":"94ce740235742e994268234c0b7c6ac490ef8057420dfc8498872c9ccf7067c6"}
Jul 17 13:15:11.610951 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:11.610929    2565 scope.go:117] "RemoveContainer" containerID="94ce740235742e994268234c0b7c6ac490ef8057420dfc8498872c9ccf7067c6"
Jul 17 13:15:12.618360 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:12.618329    2565 generic.go:356] "Generic (PLEG): container finished" podID="29cd9c58-050a-4766-b1e3-1ce15cdfefe4" containerID="9e9993c83dd1159b189e9cefbb87e115bd46b7609278b6c002e27acf5f4499cd" exitCode=6
Jul 17 13:15:12.618732 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:12.618412    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs" event={"ID":"29cd9c58-050a-4766-b1e3-1ce15cdfefe4","Type":"ContainerDied","Data":"9e9993c83dd1159b189e9cefbb87e115bd46b7609278b6c002e27acf5f4499cd"}
Jul 17 13:15:12.618732 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:12.618471    2565 scope.go:117] "RemoveContainer" containerID="94ce740235742e994268234c0b7c6ac490ef8057420dfc8498872c9ccf7067c6"
Jul 17 13:15:12.618732 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:12.618619    2565 scope.go:117] "RemoveContainer" containerID="9e9993c83dd1159b189e9cefbb87e115bd46b7609278b6c002e27acf5f4499cd"
Jul 17 13:15:12.618899 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:15:12.618776    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs_odh-ai-gateway-infra(29cd9c58-050a-4766-b1e3-1ce15cdfefe4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs" podUID="29cd9c58-050a-4766-b1e3-1ce15cdfefe4"
Jul 17 13:15:13.625739 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:13.625713    2565 scope.go:117] "RemoveContainer" containerID="9e9993c83dd1159b189e9cefbb87e115bd46b7609278b6c002e27acf5f4499cd"
Jul 17 13:15:13.626112 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:15:13.625864    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs_odh-ai-gateway-infra(29cd9c58-050a-4766-b1e3-1ce15cdfefe4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs" podUID="29cd9c58-050a-4766-b1e3-1ce15cdfefe4"
Jul 17 13:15:27.397266 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:27.397240    2565 scope.go:117] "RemoveContainer" containerID="9e9993c83dd1159b189e9cefbb87e115bd46b7609278b6c002e27acf5f4499cd"
Jul 17 13:15:27.704802 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:27.704742    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs" event={"ID":"29cd9c58-050a-4766-b1e3-1ce15cdfefe4","Type":"ContainerStarted","Data":"a95e70b9734ca5f3ffff1c6a4f641d4e5396c9a7d1bcc10b9eb4005ea617fc98"}
Jul 17 13:15:28.712369 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:28.712336    2565 generic.go:356] "Generic (PLEG): container finished" podID="29cd9c58-050a-4766-b1e3-1ce15cdfefe4" containerID="a95e70b9734ca5f3ffff1c6a4f641d4e5396c9a7d1bcc10b9eb4005ea617fc98" exitCode=6
Jul 17 13:15:28.712809 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:28.712380    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs" event={"ID":"29cd9c58-050a-4766-b1e3-1ce15cdfefe4","Type":"ContainerDied","Data":"a95e70b9734ca5f3ffff1c6a4f641d4e5396c9a7d1bcc10b9eb4005ea617fc98"}
Jul 17 13:15:28.712809 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:28.712424    2565 scope.go:117] "RemoveContainer" containerID="9e9993c83dd1159b189e9cefbb87e115bd46b7609278b6c002e27acf5f4499cd"
Jul 17 13:15:28.712809 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:28.712670    2565 scope.go:117] "RemoveContainer" containerID="a95e70b9734ca5f3ffff1c6a4f641d4e5396c9a7d1bcc10b9eb4005ea617fc98"
Jul 17 13:15:28.712920 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:15:28.712901    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs_odh-ai-gateway-infra(29cd9c58-050a-4766-b1e3-1ce15cdfefe4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs" podUID="29cd9c58-050a-4766-b1e3-1ce15cdfefe4"
Jul 17 13:15:28.741505 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:28.741482    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs"]
Jul 17 13:15:29.861300 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:29.861275    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs"
Jul 17 13:15:29.966923 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:29.966890    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29cd9c58-050a-4766-b1e3-1ce15cdfefe4-maas-api-service-ca\") pod \"29cd9c58-050a-4766-b1e3-1ce15cdfefe4\" (UID: \"29cd9c58-050a-4766-b1e3-1ce15cdfefe4\") "
Jul 17 13:15:29.967212 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:29.967192    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29cd9c58-050a-4766-b1e3-1ce15cdfefe4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "29cd9c58-050a-4766-b1e3-1ce15cdfefe4" (UID: "29cd9c58-050a-4766-b1e3-1ce15cdfefe4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:15:30.068160 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:30.068133    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29cd9c58-050a-4766-b1e3-1ce15cdfefe4-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:15:30.726966 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:30.726883    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs" event={"ID":"29cd9c58-050a-4766-b1e3-1ce15cdfefe4","Type":"ContainerDied","Data":"9143b393b1c4b5e1f5c480519563b7ccf429abd81e6072fb359cbe45026f44ec"}
Jul 17 13:15:30.726966 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:30.726936    2565 scope.go:117] "RemoveContainer" containerID="a95e70b9734ca5f3ffff1c6a4f641d4e5396c9a7d1bcc10b9eb4005ea617fc98"
Jul 17 13:15:30.727200 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:30.726892    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs"
Jul 17 13:15:30.753236 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:30.753211    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs"]
Jul 17 13:15:30.764857 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:30.764837    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-k85gs"]
Jul 17 13:15:32.403836 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:15:32.403799    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29cd9c58-050a-4766-b1e3-1ce15cdfefe4" path="/var/lib/kubelet/pods/29cd9c58-050a-4766-b1e3-1ce15cdfefe4/volumes"
Jul 17 13:16:07.367437 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:07.367408    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:16:07.457925 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:07.457891    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:16:09.685906 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:09.685872    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7"]
Jul 17 13:16:09.686374 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:09.686359    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29cd9c58-050a-4766-b1e3-1ce15cdfefe4" containerName="revoke-keys"
Jul 17 13:16:09.686419 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:09.686376    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="29cd9c58-050a-4766-b1e3-1ce15cdfefe4" containerName="revoke-keys"
Jul 17 13:16:09.686419 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:09.686392    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29cd9c58-050a-4766-b1e3-1ce15cdfefe4" containerName="revoke-keys"
Jul 17 13:16:09.686419 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:09.686398    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="29cd9c58-050a-4766-b1e3-1ce15cdfefe4" containerName="revoke-keys"
Jul 17 13:16:09.686419 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:09.686413    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5643137-282c-4242-8ce7-518ebf5945d6" containerName="revoke-keys"
Jul 17 13:16:09.686419 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:09.686419    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5643137-282c-4242-8ce7-518ebf5945d6" containerName="revoke-keys"
Jul 17 13:16:09.686574 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:09.686480    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="29cd9c58-050a-4766-b1e3-1ce15cdfefe4" containerName="revoke-keys"
Jul 17 13:16:09.686574 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:09.686487    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="29cd9c58-050a-4766-b1e3-1ce15cdfefe4" containerName="revoke-keys"
Jul 17 13:16:09.686574 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:09.686494    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="29cd9c58-050a-4766-b1e3-1ce15cdfefe4" containerName="revoke-keys"
Jul 17 13:16:09.689498 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:09.689481    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7"
Jul 17 13:16:09.692491 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:09.692468    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:16:09.698249 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:09.698214    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7"]
Jul 17 13:16:09.730551 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:09.730530    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/199a5cc9-c171-4301-9bb9-104a27fd20a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7\" (UID: \"199a5cc9-c171-4301-9bb9-104a27fd20a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7"
Jul 17 13:16:09.831153 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:09.831124    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/199a5cc9-c171-4301-9bb9-104a27fd20a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7\" (UID: \"199a5cc9-c171-4301-9bb9-104a27fd20a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7"
Jul 17 13:16:09.831766 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:09.831749    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/199a5cc9-c171-4301-9bb9-104a27fd20a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7\" (UID: \"199a5cc9-c171-4301-9bb9-104a27fd20a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7"
Jul 17 13:16:10.001909 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:10.001838    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7"
Jul 17 13:16:10.139341 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:10.139301    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7"]
Jul 17 13:16:10.144147 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:16:10.144117    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199a5cc9_c171_4301_9bb9_104a27fd20a9.slice/crio-fb32f372cc428a35f5256a9670f50610ddeeef2867ec85dc33b130c6f802249a WatchSource:0}: Error finding container fb32f372cc428a35f5256a9670f50610ddeeef2867ec85dc33b130c6f802249a: Status 404 returned error can't find the container with id fb32f372cc428a35f5256a9670f50610ddeeef2867ec85dc33b130c6f802249a
Jul 17 13:16:10.960271 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:10.960240    2565 generic.go:356] "Generic (PLEG): container finished" podID="199a5cc9-c171-4301-9bb9-104a27fd20a9" containerID="fbf6ed2f349ca7054a2bba9717781d939f5b6e2b90a7dd010346a7ea1d96efa1" exitCode=6
Jul 17 13:16:10.960688 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:10.960332    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7" event={"ID":"199a5cc9-c171-4301-9bb9-104a27fd20a9","Type":"ContainerDied","Data":"fbf6ed2f349ca7054a2bba9717781d939f5b6e2b90a7dd010346a7ea1d96efa1"}
Jul 17 13:16:10.960688 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:10.960396    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7" event={"ID":"199a5cc9-c171-4301-9bb9-104a27fd20a9","Type":"ContainerStarted","Data":"fb32f372cc428a35f5256a9670f50610ddeeef2867ec85dc33b130c6f802249a"}
Jul 17 13:16:10.960688 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:10.960469    2565 scope.go:117] "RemoveContainer" containerID="fbf6ed2f349ca7054a2bba9717781d939f5b6e2b90a7dd010346a7ea1d96efa1"
Jul 17 13:16:11.967820 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:11.967731    2565 generic.go:356] "Generic (PLEG): container finished" podID="199a5cc9-c171-4301-9bb9-104a27fd20a9" containerID="44f5120eaf402b553deaf770960bdfb56848a65e6f8f2569ddcf850371234ed7" exitCode=6
Jul 17 13:16:11.967820 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:11.967811    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7" event={"ID":"199a5cc9-c171-4301-9bb9-104a27fd20a9","Type":"ContainerDied","Data":"44f5120eaf402b553deaf770960bdfb56848a65e6f8f2569ddcf850371234ed7"}
Jul 17 13:16:11.968356 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:11.967872    2565 scope.go:117] "RemoveContainer" containerID="fbf6ed2f349ca7054a2bba9717781d939f5b6e2b90a7dd010346a7ea1d96efa1"
Jul 17 13:16:11.968356 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:11.968085    2565 scope.go:117] "RemoveContainer" containerID="44f5120eaf402b553deaf770960bdfb56848a65e6f8f2569ddcf850371234ed7"
Jul 17 13:16:11.968356 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:16:11.968315    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7_odh-ai-gateway-infra(199a5cc9-c171-4301-9bb9-104a27fd20a9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7" podUID="199a5cc9-c171-4301-9bb9-104a27fd20a9"
Jul 17 13:16:12.975350 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:12.975327    2565 scope.go:117] "RemoveContainer" containerID="44f5120eaf402b553deaf770960bdfb56848a65e6f8f2569ddcf850371234ed7"
Jul 17 13:16:12.975723 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:16:12.975489    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7_odh-ai-gateway-infra(199a5cc9-c171-4301-9bb9-104a27fd20a9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7" podUID="199a5cc9-c171-4301-9bb9-104a27fd20a9"
Jul 17 13:16:20.226885 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:20.226853    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:16:20.328971 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:20.328937    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:16:24.397311 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:24.397283    2565 scope.go:117] "RemoveContainer" containerID="44f5120eaf402b553deaf770960bdfb56848a65e6f8f2569ddcf850371234ed7"
Jul 17 13:16:25.048000 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:25.047957    2565 generic.go:356] "Generic (PLEG): container finished" podID="199a5cc9-c171-4301-9bb9-104a27fd20a9" containerID="f1d50c3545d9a71152c1ff781ec9444d9f212de2c699876aa3cebf039e46609e" exitCode=6
Jul 17 13:16:25.048167 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:25.048014    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7" event={"ID":"199a5cc9-c171-4301-9bb9-104a27fd20a9","Type":"ContainerDied","Data":"f1d50c3545d9a71152c1ff781ec9444d9f212de2c699876aa3cebf039e46609e"}
Jul 17 13:16:25.048167 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:25.048079    2565 scope.go:117] "RemoveContainer" containerID="44f5120eaf402b553deaf770960bdfb56848a65e6f8f2569ddcf850371234ed7"
Jul 17 13:16:25.048390 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:25.048374    2565 scope.go:117] "RemoveContainer" containerID="f1d50c3545d9a71152c1ff781ec9444d9f212de2c699876aa3cebf039e46609e"
Jul 17 13:16:25.048664 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:16:25.048627    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7_odh-ai-gateway-infra(199a5cc9-c171-4301-9bb9-104a27fd20a9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7" podUID="199a5cc9-c171-4301-9bb9-104a27fd20a9"
Jul 17 13:16:26.079597 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:26.079564    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7"]
Jul 17 13:16:26.227120 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:26.227096    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7"
Jul 17 13:16:26.280489 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:26.280454    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/199a5cc9-c171-4301-9bb9-104a27fd20a9-maas-api-service-ca\") pod \"199a5cc9-c171-4301-9bb9-104a27fd20a9\" (UID: \"199a5cc9-c171-4301-9bb9-104a27fd20a9\") "
Jul 17 13:16:26.280809 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:26.280787    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/199a5cc9-c171-4301-9bb9-104a27fd20a9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "199a5cc9-c171-4301-9bb9-104a27fd20a9" (UID: "199a5cc9-c171-4301-9bb9-104a27fd20a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:16:26.381376 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:26.381286    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/199a5cc9-c171-4301-9bb9-104a27fd20a9-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:16:27.063386 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:27.063351    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7" event={"ID":"199a5cc9-c171-4301-9bb9-104a27fd20a9","Type":"ContainerDied","Data":"fb32f372cc428a35f5256a9670f50610ddeeef2867ec85dc33b130c6f802249a"}
Jul 17 13:16:27.063386 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:27.063378    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7"
Jul 17 13:16:27.063639 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:27.063420    2565 scope.go:117] "RemoveContainer" containerID="f1d50c3545d9a71152c1ff781ec9444d9f212de2c699876aa3cebf039e46609e"
Jul 17 13:16:27.083943 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:27.083916    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7"]
Jul 17 13:16:27.089840 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:27.089815    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-z47z7"]
Jul 17 13:16:28.406052 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:28.406011    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="199a5cc9-c171-4301-9bb9-104a27fd20a9" path="/var/lib/kubelet/pods/199a5cc9-c171-4301-9bb9-104a27fd20a9/volumes"
Jul 17 13:16:38.335253 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:38.335205    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:16:38.433113 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:38.433078    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:16:44.449467 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:44.449430    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:16:44.714086 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:44.713995    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:16:44.959909 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:44.959874    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:16:50.602657 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:50.602624    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:16:50.674787 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:50.674752    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:16:51.112312 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:51.112275    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:16:51.214659 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:51.214628    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:16:51.313539 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:51.313496    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:16:52.780991 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:52.780955    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:16:55.456803 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:55.456773    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:16:58.648141 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:58.648103    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:16:58.753676 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:58.753639    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:16:58.859797 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:58.859764    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:16:58.948604 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:58.948516    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:16:59.393735 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:16:59.393703    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:17:25.387761 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:25.387726    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:17:31.390931 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:31.390892    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:17:31.496697 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:31.496655    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:17:31.596369 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:31.596340    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:17:36.686486 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:36.686458    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 13:17:36.692072 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:36.692045    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 13:17:36.699966 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:36.699938    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 13:17:36.705520 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:36.705494    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 13:17:47.477281 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:47.477243    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:17:53.412602 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:53.412564    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:17:53.504024 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:53.503974    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:17:53.583509 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:53.583470    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:17:58.439459 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:58.439365    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g"]
Jul 17 13:17:58.440079 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:58.440048    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="199a5cc9-c171-4301-9bb9-104a27fd20a9" containerName="revoke-keys"
Jul 17 13:17:58.440079 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:58.440069    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="199a5cc9-c171-4301-9bb9-104a27fd20a9" containerName="revoke-keys"
Jul 17 13:17:58.440216 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:58.440086    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29cd9c58-050a-4766-b1e3-1ce15cdfefe4" containerName="revoke-keys"
Jul 17 13:17:58.440216 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:58.440095    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="29cd9c58-050a-4766-b1e3-1ce15cdfefe4" containerName="revoke-keys"
Jul 17 13:17:58.440216 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:58.440121    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="199a5cc9-c171-4301-9bb9-104a27fd20a9" containerName="revoke-keys"
Jul 17 13:17:58.440216 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:58.440131    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="199a5cc9-c171-4301-9bb9-104a27fd20a9" containerName="revoke-keys"
Jul 17 13:17:58.440216 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:58.440141    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="199a5cc9-c171-4301-9bb9-104a27fd20a9" containerName="revoke-keys"
Jul 17 13:17:58.440216 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:58.440149    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="199a5cc9-c171-4301-9bb9-104a27fd20a9" containerName="revoke-keys"
Jul 17 13:17:58.440503 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:58.440273    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="199a5cc9-c171-4301-9bb9-104a27fd20a9" containerName="revoke-keys"
Jul 17 13:17:58.440503 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:58.440286    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="199a5cc9-c171-4301-9bb9-104a27fd20a9" containerName="revoke-keys"
Jul 17 13:17:58.440503 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:58.440300    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="199a5cc9-c171-4301-9bb9-104a27fd20a9" containerName="revoke-keys"
Jul 17 13:17:58.443444 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:58.443423    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g"
Jul 17 13:17:58.446493 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:58.446469    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:17:58.450116 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:58.450087    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g"]
Jul 17 13:17:58.485870 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:58.485831    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99527600-b0b0-4850-96af-027957ccdd18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-v227g\" (UID: \"99527600-b0b0-4850-96af-027957ccdd18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g"
Jul 17 13:17:58.586708 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:58.586666    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99527600-b0b0-4850-96af-027957ccdd18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-v227g\" (UID: \"99527600-b0b0-4850-96af-027957ccdd18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g"
Jul 17 13:17:58.587329 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:58.587310    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99527600-b0b0-4850-96af-027957ccdd18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-v227g\" (UID: \"99527600-b0b0-4850-96af-027957ccdd18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g"
Jul 17 13:17:58.757188 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:58.757092    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g"
Jul 17 13:17:58.901505 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:58.901475    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g"]
Jul 17 13:17:58.902377 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:17:58.902352    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99527600_b0b0_4850_96af_027957ccdd18.slice/crio-23b78ac1464a77c80591b5d136276f80aaa763256c8d1f56a5c49adfb6887fca WatchSource:0}: Error finding container 23b78ac1464a77c80591b5d136276f80aaa763256c8d1f56a5c49adfb6887fca: Status 404 returned error can't find the container with id 23b78ac1464a77c80591b5d136276f80aaa763256c8d1f56a5c49adfb6887fca
Jul 17 13:17:59.616317 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:59.616285    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g" event={"ID":"99527600-b0b0-4850-96af-027957ccdd18","Type":"ContainerStarted","Data":"fe06548695d1124792a44e0bd05935294a3aede03a908d66dcc5d769b5fba8a1"}
Jul 17 13:17:59.616689 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:59.616349    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g" event={"ID":"99527600-b0b0-4850-96af-027957ccdd18","Type":"ContainerStarted","Data":"23b78ac1464a77c80591b5d136276f80aaa763256c8d1f56a5c49adfb6887fca"}
Jul 17 13:17:59.631741 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:17:59.631695    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g" podStartSLOduration=1.631677534 podStartE2EDuration="1.631677534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:17:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:17:59.631161162 +0000 UTC m=+3323.762545824" watchObservedRunningTime="2026-07-17 13:17:59.631677534 +0000 UTC m=+3323.763062194"
Jul 17 13:18:00.624023 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:00.623968    2565 generic.go:356] "Generic (PLEG): container finished" podID="99527600-b0b0-4850-96af-027957ccdd18" containerID="fe06548695d1124792a44e0bd05935294a3aede03a908d66dcc5d769b5fba8a1" exitCode=6
Jul 17 13:18:00.624428 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:00.624030    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g" event={"ID":"99527600-b0b0-4850-96af-027957ccdd18","Type":"ContainerDied","Data":"fe06548695d1124792a44e0bd05935294a3aede03a908d66dcc5d769b5fba8a1"}
Jul 17 13:18:00.624428 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:00.624318    2565 scope.go:117] "RemoveContainer" containerID="fe06548695d1124792a44e0bd05935294a3aede03a908d66dcc5d769b5fba8a1"
Jul 17 13:18:01.631909 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:01.631867    2565 generic.go:356] "Generic (PLEG): container finished" podID="99527600-b0b0-4850-96af-027957ccdd18" containerID="14eae331a630fbcad3bc942ec13a4fc428839975a54aedd11e129136e99bb879" exitCode=6
Jul 17 13:18:01.632343 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:01.631920    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g" event={"ID":"99527600-b0b0-4850-96af-027957ccdd18","Type":"ContainerDied","Data":"14eae331a630fbcad3bc942ec13a4fc428839975a54aedd11e129136e99bb879"}
Jul 17 13:18:01.632343 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:01.631977    2565 scope.go:117] "RemoveContainer" containerID="fe06548695d1124792a44e0bd05935294a3aede03a908d66dcc5d769b5fba8a1"
Jul 17 13:18:01.632343 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:01.632185    2565 scope.go:117] "RemoveContainer" containerID="14eae331a630fbcad3bc942ec13a4fc428839975a54aedd11e129136e99bb879"
Jul 17 13:18:01.632461 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:18:01.632412    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-v227g_odh-ai-gateway-infra(99527600-b0b0-4850-96af-027957ccdd18)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g" podUID="99527600-b0b0-4850-96af-027957ccdd18"
Jul 17 13:18:02.639869 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:02.639843    2565 scope.go:117] "RemoveContainer" containerID="14eae331a630fbcad3bc942ec13a4fc428839975a54aedd11e129136e99bb879"
Jul 17 13:18:02.640290 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:18:02.640065    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-v227g_odh-ai-gateway-infra(99527600-b0b0-4850-96af-027957ccdd18)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g" podUID="99527600-b0b0-4850-96af-027957ccdd18"
Jul 17 13:18:04.183127 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:04.183086    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:04.994360 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:04.994326    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:10.785753 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:10.785702    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:10.893179 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:10.893144    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:10.988095 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:10.988056    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:11.085994 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:11.085946    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:11.844942 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:11.844910    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:11.943634 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:11.943603    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:12.045852 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:12.045817    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:13.678266 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:13.678226    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:15.397356 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:15.397321    2565 scope.go:117] "RemoveContainer" containerID="14eae331a630fbcad3bc942ec13a4fc428839975a54aedd11e129136e99bb879"
Jul 17 13:18:16.725454 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:16.725421    2565 generic.go:356] "Generic (PLEG): container finished" podID="99527600-b0b0-4850-96af-027957ccdd18" containerID="0087253d7854ef8051b0d43a4705191facafc0d80b42f669757f68e4f1ee9690" exitCode=6
Jul 17 13:18:16.725834 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:16.725491    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g" event={"ID":"99527600-b0b0-4850-96af-027957ccdd18","Type":"ContainerDied","Data":"0087253d7854ef8051b0d43a4705191facafc0d80b42f669757f68e4f1ee9690"}
Jul 17 13:18:16.725834 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:16.725555    2565 scope.go:117] "RemoveContainer" containerID="14eae331a630fbcad3bc942ec13a4fc428839975a54aedd11e129136e99bb879"
Jul 17 13:18:16.725996 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:16.725843    2565 scope.go:117] "RemoveContainer" containerID="0087253d7854ef8051b0d43a4705191facafc0d80b42f669757f68e4f1ee9690"
Jul 17 13:18:16.726102 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:18:16.726081    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-v227g_odh-ai-gateway-infra(99527600-b0b0-4850-96af-027957ccdd18)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g" podUID="99527600-b0b0-4850-96af-027957ccdd18"
Jul 17 13:18:17.751892 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:17.751862    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g"]
Jul 17 13:18:17.900664 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:17.900641    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g"
Jul 17 13:18:17.941467 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:17.941432    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99527600-b0b0-4850-96af-027957ccdd18-maas-api-service-ca\") pod \"99527600-b0b0-4850-96af-027957ccdd18\" (UID: \"99527600-b0b0-4850-96af-027957ccdd18\") "
Jul 17 13:18:17.941784 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:17.941760    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99527600-b0b0-4850-96af-027957ccdd18-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "99527600-b0b0-4850-96af-027957ccdd18" (UID: "99527600-b0b0-4850-96af-027957ccdd18"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:18:18.043043 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:18.042907    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99527600-b0b0-4850-96af-027957ccdd18-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:18:18.741804 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:18.741714    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g"
Jul 17 13:18:18.741804 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:18.741756    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g" event={"ID":"99527600-b0b0-4850-96af-027957ccdd18","Type":"ContainerDied","Data":"23b78ac1464a77c80591b5d136276f80aaa763256c8d1f56a5c49adfb6887fca"}
Jul 17 13:18:18.742095 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:18.741815    2565 scope.go:117] "RemoveContainer" containerID="0087253d7854ef8051b0d43a4705191facafc0d80b42f669757f68e4f1ee9690"
Jul 17 13:18:18.760165 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:18.760135    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g"]
Jul 17 13:18:18.765886 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:18.765860    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-v227g"]
Jul 17 13:18:20.060416 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:20.060380    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:20.156905 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:20.156875    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:20.403410 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:20.403328    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99527600-b0b0-4850-96af-027957ccdd18" path="/var/lib/kubelet/pods/99527600-b0b0-4850-96af-027957ccdd18/volumes"
Jul 17 13:18:20.715208 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:20.715124    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:26.540220 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:26.540175    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:26.654820 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:26.654789    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:26.797447 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:26.797414    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:27.170358 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:27.170250    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:27.234567 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:27.234534    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:30.141777 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:30.141748    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:30.238895 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:30.238863    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:30.438091 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:30.437972    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:30.769492 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:30.769403    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:30.976274 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:30.976229    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:31.079285 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:31.079247    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:34.624416 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:34.624379    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:35.297311 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:35.297267    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:35.381749 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:35.381717    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:35.728290 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:35.728191    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:39.812213 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:39.812172    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b"]
Jul 17 13:18:39.812690 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:39.812612    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99527600-b0b0-4850-96af-027957ccdd18" containerName="revoke-keys"
Jul 17 13:18:39.812690 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:39.812623    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="99527600-b0b0-4850-96af-027957ccdd18" containerName="revoke-keys"
Jul 17 13:18:39.812690 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:39.812641    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99527600-b0b0-4850-96af-027957ccdd18" containerName="revoke-keys"
Jul 17 13:18:39.812690 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:39.812646    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="99527600-b0b0-4850-96af-027957ccdd18" containerName="revoke-keys"
Jul 17 13:18:39.812690 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:39.812652    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99527600-b0b0-4850-96af-027957ccdd18" containerName="revoke-keys"
Jul 17 13:18:39.812690 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:39.812658    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="99527600-b0b0-4850-96af-027957ccdd18" containerName="revoke-keys"
Jul 17 13:18:39.813015 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:39.812721    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="99527600-b0b0-4850-96af-027957ccdd18" containerName="revoke-keys"
Jul 17 13:18:39.813015 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:39.812730    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="99527600-b0b0-4850-96af-027957ccdd18" containerName="revoke-keys"
Jul 17 13:18:39.817309 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:39.817292    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b"
Jul 17 13:18:39.820401 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:39.820370    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:18:39.823802 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:39.823772    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b"]
Jul 17 13:18:39.954830 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:39.954786    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25f344fa-9f3e-4b72-8c43-438b9c1f3f53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b\" (UID: \"25f344fa-9f3e-4b72-8c43-438b9c1f3f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b"
Jul 17 13:18:40.056134 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:40.056095    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25f344fa-9f3e-4b72-8c43-438b9c1f3f53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b\" (UID: \"25f344fa-9f3e-4b72-8c43-438b9c1f3f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b"
Jul 17 13:18:40.056864 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:40.056841    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25f344fa-9f3e-4b72-8c43-438b9c1f3f53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b\" (UID: \"25f344fa-9f3e-4b72-8c43-438b9c1f3f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b"
Jul 17 13:18:40.130842 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:40.130772    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b"
Jul 17 13:18:40.274790 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:40.274764    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b"]
Jul 17 13:18:40.276212 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:18:40.276187    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f344fa_9f3e_4b72_8c43_438b9c1f3f53.slice/crio-af93ce601ae98b15e75285d1738e4f638eabcd54e0a49991d767cf960593d902 WatchSource:0}: Error finding container af93ce601ae98b15e75285d1738e4f638eabcd54e0a49991d767cf960593d902: Status 404 returned error can't find the container with id af93ce601ae98b15e75285d1738e4f638eabcd54e0a49991d767cf960593d902
Jul 17 13:18:40.876638 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:40.876602    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b" event={"ID":"25f344fa-9f3e-4b72-8c43-438b9c1f3f53","Type":"ContainerStarted","Data":"d00ac56456f06e3b2e76be1080a4cb677703cc54bda0af66ac9703bb59681075"}
Jul 17 13:18:40.877093 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:40.876663    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b" event={"ID":"25f344fa-9f3e-4b72-8c43-438b9c1f3f53","Type":"ContainerStarted","Data":"af93ce601ae98b15e75285d1738e4f638eabcd54e0a49991d767cf960593d902"}
Jul 17 13:18:40.900058 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:40.896397    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b" podStartSLOduration=1.896375111 podStartE2EDuration="1.896375111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:18:40.89303029 +0000 UTC m=+3365.024414979" watchObservedRunningTime="2026-07-17 13:18:40.896375111 +0000 UTC m=+3365.027759774"
Jul 17 13:18:41.557208 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:41.557173    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:41.663581 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:41.663546    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:41.885103 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:41.885018    2565 generic.go:356] "Generic (PLEG): container finished" podID="25f344fa-9f3e-4b72-8c43-438b9c1f3f53" containerID="d00ac56456f06e3b2e76be1080a4cb677703cc54bda0af66ac9703bb59681075" exitCode=6
Jul 17 13:18:41.885515 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:41.885105    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b" event={"ID":"25f344fa-9f3e-4b72-8c43-438b9c1f3f53","Type":"ContainerDied","Data":"d00ac56456f06e3b2e76be1080a4cb677703cc54bda0af66ac9703bb59681075"}
Jul 17 13:18:41.885515 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:41.885275    2565 scope.go:117] "RemoveContainer" containerID="d00ac56456f06e3b2e76be1080a4cb677703cc54bda0af66ac9703bb59681075"
Jul 17 13:18:42.892428 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:42.892397    2565 generic.go:356] "Generic (PLEG): container finished" podID="25f344fa-9f3e-4b72-8c43-438b9c1f3f53" containerID="041b28301fa0db67cc2671ffe1d50cffcd3c143b556248708cf115193378a89e" exitCode=6
Jul 17 13:18:42.892863 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:42.892485    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b" event={"ID":"25f344fa-9f3e-4b72-8c43-438b9c1f3f53","Type":"ContainerDied","Data":"041b28301fa0db67cc2671ffe1d50cffcd3c143b556248708cf115193378a89e"}
Jul 17 13:18:42.892863 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:42.892541    2565 scope.go:117] "RemoveContainer" containerID="d00ac56456f06e3b2e76be1080a4cb677703cc54bda0af66ac9703bb59681075"
Jul 17 13:18:42.892863 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:42.892643    2565 scope.go:117] "RemoveContainer" containerID="041b28301fa0db67cc2671ffe1d50cffcd3c143b556248708cf115193378a89e"
Jul 17 13:18:42.893023 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:18:42.892892    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b_odh-ai-gateway-infra(25f344fa-9f3e-4b72-8c43-438b9c1f3f53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b" podUID="25f344fa-9f3e-4b72-8c43-438b9c1f3f53"
Jul 17 13:18:43.900940 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:43.900916    2565 scope.go:117] "RemoveContainer" containerID="041b28301fa0db67cc2671ffe1d50cffcd3c143b556248708cf115193378a89e"
Jul 17 13:18:43.901347 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:18:43.901148    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b_odh-ai-gateway-infra(25f344fa-9f3e-4b72-8c43-438b9c1f3f53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b" podUID="25f344fa-9f3e-4b72-8c43-438b9c1f3f53"
Jul 17 13:18:52.532853 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:52.532822    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:57.020369 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:57.020332    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:18:58.397090 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:58.397063    2565 scope.go:117] "RemoveContainer" containerID="041b28301fa0db67cc2671ffe1d50cffcd3c143b556248708cf115193378a89e"
Jul 17 13:18:58.994455 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:58.994377    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b" event={"ID":"25f344fa-9f3e-4b72-8c43-438b9c1f3f53","Type":"ContainerStarted","Data":"c13e58acab467a8c70e7036c0a9dc7bce94fdcd7c256e7c832bfba39fef127e5"}
Jul 17 13:18:59.057717 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:18:59.057669    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:00.001881 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:00.001849    2565 generic.go:356] "Generic (PLEG): container finished" podID="25f344fa-9f3e-4b72-8c43-438b9c1f3f53" containerID="c13e58acab467a8c70e7036c0a9dc7bce94fdcd7c256e7c832bfba39fef127e5" exitCode=6
Jul 17 13:19:00.002342 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:00.001937    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b" event={"ID":"25f344fa-9f3e-4b72-8c43-438b9c1f3f53","Type":"ContainerDied","Data":"c13e58acab467a8c70e7036c0a9dc7bce94fdcd7c256e7c832bfba39fef127e5"}
Jul 17 13:19:00.002342 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:00.002029    2565 scope.go:117] "RemoveContainer" containerID="041b28301fa0db67cc2671ffe1d50cffcd3c143b556248708cf115193378a89e"
Jul 17 13:19:00.002342 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:00.002303    2565 scope.go:117] "RemoveContainer" containerID="c13e58acab467a8c70e7036c0a9dc7bce94fdcd7c256e7c832bfba39fef127e5"
Jul 17 13:19:00.002547 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:19:00.002528    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b_odh-ai-gateway-infra(25f344fa-9f3e-4b72-8c43-438b9c1f3f53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b" podUID="25f344fa-9f3e-4b72-8c43-438b9c1f3f53"
Jul 17 13:19:00.025927 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:00.025904    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b"]
Jul 17 13:19:01.152476 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:01.152453    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b"
Jul 17 13:19:01.243335 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:01.243313    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25f344fa-9f3e-4b72-8c43-438b9c1f3f53-maas-api-service-ca\") pod \"25f344fa-9f3e-4b72-8c43-438b9c1f3f53\" (UID: \"25f344fa-9f3e-4b72-8c43-438b9c1f3f53\") "
Jul 17 13:19:01.243656 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:01.243635    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25f344fa-9f3e-4b72-8c43-438b9c1f3f53-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "25f344fa-9f3e-4b72-8c43-438b9c1f3f53" (UID: "25f344fa-9f3e-4b72-8c43-438b9c1f3f53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:19:01.344105 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:01.344079    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25f344fa-9f3e-4b72-8c43-438b9c1f3f53-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:19:02.018061 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:02.018029    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b"
Jul 17 13:19:02.018061 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:02.018044    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b" event={"ID":"25f344fa-9f3e-4b72-8c43-438b9c1f3f53","Type":"ContainerDied","Data":"af93ce601ae98b15e75285d1738e4f638eabcd54e0a49991d767cf960593d902"}
Jul 17 13:19:02.018308 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:02.018107    2565 scope.go:117] "RemoveContainer" containerID="c13e58acab467a8c70e7036c0a9dc7bce94fdcd7c256e7c832bfba39fef127e5"
Jul 17 13:19:02.057399 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:02.057362    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b"]
Jul 17 13:19:02.074969 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:02.074946    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-p7z5b"]
Jul 17 13:19:02.403214 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:02.403182    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25f344fa-9f3e-4b72-8c43-438b9c1f3f53" path="/var/lib/kubelet/pods/25f344fa-9f3e-4b72-8c43-438b9c1f3f53/volumes"
Jul 17 13:19:03.205279 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:03.205245    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:03.275321 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:03.275293    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:07.691468 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:07.691432    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:07.922293 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:07.922257    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:08.073015 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:08.072969    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:08.182285 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:08.182254    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:08.368570 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:08.368495    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:14.596745 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:14.596709    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:14.708948 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:14.708919    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:14.802960 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:14.802929    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:14.992202 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:14.992125    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:20.987456 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:20.987423    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:21.090672 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:21.090636    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:27.835897 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:27.835802    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:28.520266 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:28.520235    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:28.885080 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:28.885047    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:31.402355 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:31.402319    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:31.506463 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:31.506428    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:31.610633 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:31.610601    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:31.764284 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:31.764193    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:32.184159 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:32.184128    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:32.281060 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:32.281026    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:35.879875 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:35.879846    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:19:48.542960 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:48.542926    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr"]
Jul 17 13:19:48.543390 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:48.543375    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25f344fa-9f3e-4b72-8c43-438b9c1f3f53" containerName="revoke-keys"
Jul 17 13:19:48.543434 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:48.543392    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="25f344fa-9f3e-4b72-8c43-438b9c1f3f53" containerName="revoke-keys"
Jul 17 13:19:48.543434 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:48.543404    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25f344fa-9f3e-4b72-8c43-438b9c1f3f53" containerName="revoke-keys"
Jul 17 13:19:48.543434 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:48.543410    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="25f344fa-9f3e-4b72-8c43-438b9c1f3f53" containerName="revoke-keys"
Jul 17 13:19:48.543530 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:48.543490    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="25f344fa-9f3e-4b72-8c43-438b9c1f3f53" containerName="revoke-keys"
Jul 17 13:19:48.543530 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:48.543499    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="99527600-b0b0-4850-96af-027957ccdd18" containerName="revoke-keys"
Jul 17 13:19:48.543530 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:48.543505    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="25f344fa-9f3e-4b72-8c43-438b9c1f3f53" containerName="revoke-keys"
Jul 17 13:19:48.546246 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:48.546230    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr"
Jul 17 13:19:48.549331 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:48.549317    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:19:48.551653 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:48.551576    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80574279-8f6a-4941-93fd-76f63b8c2ad1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr\" (UID: \"80574279-8f6a-4941-93fd-76f63b8c2ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr"
Jul 17 13:19:48.555561 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:48.555537    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr"]
Jul 17 13:19:48.652971 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:48.652941    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80574279-8f6a-4941-93fd-76f63b8c2ad1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr\" (UID: \"80574279-8f6a-4941-93fd-76f63b8c2ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr"
Jul 17 13:19:48.653562 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:48.653542    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80574279-8f6a-4941-93fd-76f63b8c2ad1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr\" (UID: \"80574279-8f6a-4941-93fd-76f63b8c2ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr"
Jul 17 13:19:48.857678 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:48.857644    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr"
Jul 17 13:19:49.005072 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:49.005040    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr"]
Jul 17 13:19:49.005714 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:19:49.005690    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80574279_8f6a_4941_93fd_76f63b8c2ad1.slice/crio-9a41324e9dc8f72f56dc2e37e55e007a22bbec8a511f80a6a0a05fac4a5e8504 WatchSource:0}: Error finding container 9a41324e9dc8f72f56dc2e37e55e007a22bbec8a511f80a6a0a05fac4a5e8504: Status 404 returned error can't find the container with id 9a41324e9dc8f72f56dc2e37e55e007a22bbec8a511f80a6a0a05fac4a5e8504
Jul 17 13:19:49.305039 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:49.305007    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr" event={"ID":"80574279-8f6a-4941-93fd-76f63b8c2ad1","Type":"ContainerStarted","Data":"f4bfd3ea8b3da7b7f5f69e58c510860d0fadc05491da28cd482f767ca524a72e"}
Jul 17 13:19:49.305162 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:49.305061    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr" event={"ID":"80574279-8f6a-4941-93fd-76f63b8c2ad1","Type":"ContainerStarted","Data":"9a41324e9dc8f72f56dc2e37e55e007a22bbec8a511f80a6a0a05fac4a5e8504"}
Jul 17 13:19:49.322902 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:49.322854    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr" podStartSLOduration=1.322840048 podStartE2EDuration="1.322840048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:19:49.320934583 +0000 UTC m=+3433.452319259" watchObservedRunningTime="2026-07-17 13:19:49.322840048 +0000 UTC m=+3433.454224752"
Jul 17 13:19:50.313818 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:50.313779    2565 generic.go:356] "Generic (PLEG): container finished" podID="80574279-8f6a-4941-93fd-76f63b8c2ad1" containerID="f4bfd3ea8b3da7b7f5f69e58c510860d0fadc05491da28cd482f767ca524a72e" exitCode=6
Jul 17 13:19:50.314257 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:50.313863    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr" event={"ID":"80574279-8f6a-4941-93fd-76f63b8c2ad1","Type":"ContainerDied","Data":"f4bfd3ea8b3da7b7f5f69e58c510860d0fadc05491da28cd482f767ca524a72e"}
Jul 17 13:19:50.314257 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:50.314050    2565 scope.go:117] "RemoveContainer" containerID="f4bfd3ea8b3da7b7f5f69e58c510860d0fadc05491da28cd482f767ca524a72e"
Jul 17 13:19:51.322065 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:51.322030    2565 generic.go:356] "Generic (PLEG): container finished" podID="80574279-8f6a-4941-93fd-76f63b8c2ad1" containerID="6402febf3e09fb32aebd56c1c9837ee020b7f1651688bd379e1c46252a565aee" exitCode=6
Jul 17 13:19:51.322546 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:51.322083    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr" event={"ID":"80574279-8f6a-4941-93fd-76f63b8c2ad1","Type":"ContainerDied","Data":"6402febf3e09fb32aebd56c1c9837ee020b7f1651688bd379e1c46252a565aee"}
Jul 17 13:19:51.322546 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:51.322146    2565 scope.go:117] "RemoveContainer" containerID="f4bfd3ea8b3da7b7f5f69e58c510860d0fadc05491da28cd482f767ca524a72e"
Jul 17 13:19:51.322546 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:51.322350    2565 scope.go:117] "RemoveContainer" containerID="6402febf3e09fb32aebd56c1c9837ee020b7f1651688bd379e1c46252a565aee"
Jul 17 13:19:51.322703 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:19:51.322608    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr_odh-ai-gateway-infra(80574279-8f6a-4941-93fd-76f63b8c2ad1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr" podUID="80574279-8f6a-4941-93fd-76f63b8c2ad1"
Jul 17 13:19:52.329123 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:19:52.329094    2565 scope.go:117] "RemoveContainer" containerID="6402febf3e09fb32aebd56c1c9837ee020b7f1651688bd379e1c46252a565aee"
Jul 17 13:19:52.329492 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:19:52.329263    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr_odh-ai-gateway-infra(80574279-8f6a-4941-93fd-76f63b8c2ad1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr" podUID="80574279-8f6a-4941-93fd-76f63b8c2ad1"
Jul 17 13:20:05.396363 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:05.396335    2565 scope.go:117] "RemoveContainer" containerID="6402febf3e09fb32aebd56c1c9837ee020b7f1651688bd379e1c46252a565aee"
Jul 17 13:20:06.414731 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:06.414698    2565 generic.go:356] "Generic (PLEG): container finished" podID="80574279-8f6a-4941-93fd-76f63b8c2ad1" containerID="1b48569fd226c9a7f186018a17cee549b710c4aac6924dffbdb64e990a9d7dae" exitCode=6
Jul 17 13:20:06.415159 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:06.414768    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr" event={"ID":"80574279-8f6a-4941-93fd-76f63b8c2ad1","Type":"ContainerDied","Data":"1b48569fd226c9a7f186018a17cee549b710c4aac6924dffbdb64e990a9d7dae"}
Jul 17 13:20:06.415159 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:06.414817    2565 scope.go:117] "RemoveContainer" containerID="6402febf3e09fb32aebd56c1c9837ee020b7f1651688bd379e1c46252a565aee"
Jul 17 13:20:06.415159 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:06.415110    2565 scope.go:117] "RemoveContainer" containerID="1b48569fd226c9a7f186018a17cee549b710c4aac6924dffbdb64e990a9d7dae"
Jul 17 13:20:06.415316 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:20:06.415282    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr_odh-ai-gateway-infra(80574279-8f6a-4941-93fd-76f63b8c2ad1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr" podUID="80574279-8f6a-4941-93fd-76f63b8c2ad1"
Jul 17 13:20:07.467681 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:07.467650    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr"]
Jul 17 13:20:07.606847 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:07.606823    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr"
Jul 17 13:20:07.714877 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:07.714849    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80574279-8f6a-4941-93fd-76f63b8c2ad1-maas-api-service-ca\") pod \"80574279-8f6a-4941-93fd-76f63b8c2ad1\" (UID: \"80574279-8f6a-4941-93fd-76f63b8c2ad1\") "
Jul 17 13:20:07.715294 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:07.715269    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80574279-8f6a-4941-93fd-76f63b8c2ad1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "80574279-8f6a-4941-93fd-76f63b8c2ad1" (UID: "80574279-8f6a-4941-93fd-76f63b8c2ad1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:20:07.815871 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:07.815840    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80574279-8f6a-4941-93fd-76f63b8c2ad1-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:20:08.428295 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:08.428273    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr"
Jul 17 13:20:08.428434 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:08.428327    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr" event={"ID":"80574279-8f6a-4941-93fd-76f63b8c2ad1","Type":"ContainerDied","Data":"9a41324e9dc8f72f56dc2e37e55e007a22bbec8a511f80a6a0a05fac4a5e8504"}
Jul 17 13:20:08.428434 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:08.428386    2565 scope.go:117] "RemoveContainer" containerID="1b48569fd226c9a7f186018a17cee549b710c4aac6924dffbdb64e990a9d7dae"
Jul 17 13:20:08.460277 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:08.460257    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr"]
Jul 17 13:20:08.471837 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:08.471818    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-pp6wr"]
Jul 17 13:20:10.403613 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:10.403585    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80574279-8f6a-4941-93fd-76f63b8c2ad1" path="/var/lib/kubelet/pods/80574279-8f6a-4941-93fd-76f63b8c2ad1/volumes"
Jul 17 13:20:13.712913 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:13.712850    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:20:44.932393 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:44.932358    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ad1d9a1-4w52h"]
Jul 17 13:20:44.933015 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:44.932997    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25f344fa-9f3e-4b72-8c43-438b9c1f3f53" containerName="revoke-keys"
Jul 17 13:20:44.933122 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:44.933017    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="25f344fa-9f3e-4b72-8c43-438b9c1f3f53" containerName="revoke-keys"
Jul 17 13:20:44.933122 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:44.933030    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80574279-8f6a-4941-93fd-76f63b8c2ad1" containerName="revoke-keys"
Jul 17 13:20:44.933122 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:44.933038    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="80574279-8f6a-4941-93fd-76f63b8c2ad1" containerName="revoke-keys"
Jul 17 13:20:44.933122 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:44.933051    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80574279-8f6a-4941-93fd-76f63b8c2ad1" containerName="revoke-keys"
Jul 17 13:20:44.933122 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:44.933059    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="80574279-8f6a-4941-93fd-76f63b8c2ad1" containerName="revoke-keys"
Jul 17 13:20:44.933439 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:44.933184    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="80574279-8f6a-4941-93fd-76f63b8c2ad1" containerName="revoke-keys"
Jul 17 13:20:44.933439 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:44.933198    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="25f344fa-9f3e-4b72-8c43-438b9c1f3f53" containerName="revoke-keys"
Jul 17 13:20:44.933439 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:44.933211    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="80574279-8f6a-4941-93fd-76f63b8c2ad1" containerName="revoke-keys"
Jul 17 13:20:44.933439 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:44.933221    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="80574279-8f6a-4941-93fd-76f63b8c2ad1" containerName="revoke-keys"
Jul 17 13:20:44.937518 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:44.937498    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ad1d9a1-4w52h"
Jul 17 13:20:44.940719 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:44.940515    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:20:44.944364 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:44.944342    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ad1d9a1-4w52h"]
Jul 17 13:20:45.045473 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:45.045437    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb342c5c-709a-4a95-9aa5-95114a67acaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6ad1d9a1-4w52h\" (UID: \"cb342c5c-709a-4a95-9aa5-95114a67acaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ad1d9a1-4w52h"
Jul 17 13:20:45.146774 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:45.146739    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb342c5c-709a-4a95-9aa5-95114a67acaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6ad1d9a1-4w52h\" (UID: \"cb342c5c-709a-4a95-9aa5-95114a67acaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ad1d9a1-4w52h"
Jul 17 13:20:45.147413 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:45.147390    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb342c5c-709a-4a95-9aa5-95114a67acaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6ad1d9a1-4w52h\" (UID: \"cb342c5c-709a-4a95-9aa5-95114a67acaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ad1d9a1-4w52h"
Jul 17 13:20:45.250870 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:45.250792    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ad1d9a1-4w52h"
Jul 17 13:20:45.600137 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:45.600104    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ad1d9a1-4w52h"]
Jul 17 13:20:45.601374 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:20:45.601331    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb342c5c_709a_4a95_9aa5_95114a67acaa.slice/crio-582b4e3d293600677dad657c8a7e03c726d2739691cb4ab139848aab3f40b6a2 WatchSource:0}: Error finding container 582b4e3d293600677dad657c8a7e03c726d2739691cb4ab139848aab3f40b6a2: Status 404 returned error can't find the container with id 582b4e3d293600677dad657c8a7e03c726d2739691cb4ab139848aab3f40b6a2
Jul 17 13:20:45.655232 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:45.655201    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ad1d9a1-4w52h" event={"ID":"cb342c5c-709a-4a95-9aa5-95114a67acaa","Type":"ContainerStarted","Data":"582b4e3d293600677dad657c8a7e03c726d2739691cb4ab139848aab3f40b6a2"}
Jul 17 13:20:46.662303 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:46.662220    2565 generic.go:356] "Generic (PLEG): container finished" podID="cb342c5c-709a-4a95-9aa5-95114a67acaa" containerID="15f529d0f4bf7bb666f6856a54023b209a691d139fa26a81958e42ac26f44e3d" exitCode=0
Jul 17 13:20:46.662303 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:46.662280    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ad1d9a1-4w52h" event={"ID":"cb342c5c-709a-4a95-9aa5-95114a67acaa","Type":"ContainerDied","Data":"15f529d0f4bf7bb666f6856a54023b209a691d139fa26a81958e42ac26f44e3d"}
Jul 17 13:20:47.807486 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:47.807460    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ad1d9a1-4w52h"
Jul 17 13:20:47.872762 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:47.872732    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb342c5c-709a-4a95-9aa5-95114a67acaa-maas-api-service-ca\") pod \"cb342c5c-709a-4a95-9aa5-95114a67acaa\" (UID: \"cb342c5c-709a-4a95-9aa5-95114a67acaa\") "
Jul 17 13:20:47.873133 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:47.873110    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb342c5c-709a-4a95-9aa5-95114a67acaa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cb342c5c-709a-4a95-9aa5-95114a67acaa" (UID: "cb342c5c-709a-4a95-9aa5-95114a67acaa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:20:47.974226 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:47.974156    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb342c5c-709a-4a95-9aa5-95114a67acaa-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:20:48.676316 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:48.676231    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ad1d9a1-4w52h"
Jul 17 13:20:48.676458 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:48.676232    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6ad1d9a1-4w52h" event={"ID":"cb342c5c-709a-4a95-9aa5-95114a67acaa","Type":"ContainerDied","Data":"582b4e3d293600677dad657c8a7e03c726d2739691cb4ab139848aab3f40b6a2"}
Jul 17 13:20:48.676458 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:48.676356    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="582b4e3d293600677dad657c8a7e03c726d2739691cb4ab139848aab3f40b6a2"
Jul 17 13:20:51.139252 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:51.139221    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg"]
Jul 17 13:20:51.139652 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:51.139639    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb342c5c-709a-4a95-9aa5-95114a67acaa" containerName="revoke-keys"
Jul 17 13:20:51.139696 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:51.139653    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb342c5c-709a-4a95-9aa5-95114a67acaa" containerName="revoke-keys"
Jul 17 13:20:51.139696 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:51.139674    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80574279-8f6a-4941-93fd-76f63b8c2ad1" containerName="revoke-keys"
Jul 17 13:20:51.139696 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:51.139680    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="80574279-8f6a-4941-93fd-76f63b8c2ad1" containerName="revoke-keys"
Jul 17 13:20:51.139796 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:51.139744    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb342c5c-709a-4a95-9aa5-95114a67acaa" containerName="revoke-keys"
Jul 17 13:20:51.142998 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:51.142964    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg"
Jul 17 13:20:51.146575 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:51.146558    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:20:51.156887 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:51.156856    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg"]
Jul 17 13:20:51.201271 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:51.201249    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2caf39c1-81ac-40c1-ade7-96886d5efb2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg\" (UID: \"2caf39c1-81ac-40c1-ade7-96886d5efb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg"
Jul 17 13:20:51.302654 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:51.302628    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2caf39c1-81ac-40c1-ade7-96886d5efb2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg\" (UID: \"2caf39c1-81ac-40c1-ade7-96886d5efb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg"
Jul 17 13:20:51.303281 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:51.303263    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2caf39c1-81ac-40c1-ade7-96886d5efb2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg\" (UID: \"2caf39c1-81ac-40c1-ade7-96886d5efb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg"
Jul 17 13:20:51.452651 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:51.452596    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg"
Jul 17 13:20:51.591501 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:51.591475    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg"]
Jul 17 13:20:51.592577 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:20:51.592549    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2caf39c1_81ac_40c1_ade7_96886d5efb2e.slice/crio-e7d5c5247d56789706657d0e75d3fd3833515153c9c6a58db6d2cacdfc2b8095 WatchSource:0}: Error finding container e7d5c5247d56789706657d0e75d3fd3833515153c9c6a58db6d2cacdfc2b8095: Status 404 returned error can't find the container with id e7d5c5247d56789706657d0e75d3fd3833515153c9c6a58db6d2cacdfc2b8095
Jul 17 13:20:51.696040 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:51.696010    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg" event={"ID":"2caf39c1-81ac-40c1-ade7-96886d5efb2e","Type":"ContainerStarted","Data":"e7d5c5247d56789706657d0e75d3fd3833515153c9c6a58db6d2cacdfc2b8095"}
Jul 17 13:20:52.702815 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:52.702780    2565 generic.go:356] "Generic (PLEG): container finished" podID="2caf39c1-81ac-40c1-ade7-96886d5efb2e" containerID="dc1eb6f767fdf55082f505aa5c70105552c0d3a3f7a301d584060e9b6295528f" exitCode=6
Jul 17 13:20:52.703224 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:52.702860    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg" event={"ID":"2caf39c1-81ac-40c1-ade7-96886d5efb2e","Type":"ContainerDied","Data":"dc1eb6f767fdf55082f505aa5c70105552c0d3a3f7a301d584060e9b6295528f"}
Jul 17 13:20:52.703224 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:52.703064    2565 scope.go:117] "RemoveContainer" containerID="dc1eb6f767fdf55082f505aa5c70105552c0d3a3f7a301d584060e9b6295528f"
Jul 17 13:20:53.710827 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:53.710752    2565 generic.go:356] "Generic (PLEG): container finished" podID="2caf39c1-81ac-40c1-ade7-96886d5efb2e" containerID="cc3706c21c613d8617879f2340cb50820beba0a6774060d2a21b856b089e6b17" exitCode=6
Jul 17 13:20:53.711260 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:53.710841    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg" event={"ID":"2caf39c1-81ac-40c1-ade7-96886d5efb2e","Type":"ContainerDied","Data":"cc3706c21c613d8617879f2340cb50820beba0a6774060d2a21b856b089e6b17"}
Jul 17 13:20:53.711260 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:53.710903    2565 scope.go:117] "RemoveContainer" containerID="dc1eb6f767fdf55082f505aa5c70105552c0d3a3f7a301d584060e9b6295528f"
Jul 17 13:20:53.711260 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:53.711006    2565 scope.go:117] "RemoveContainer" containerID="cc3706c21c613d8617879f2340cb50820beba0a6774060d2a21b856b089e6b17"
Jul 17 13:20:53.711260 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:20:53.711218    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg_odh-ai-gateway-infra(2caf39c1-81ac-40c1-ade7-96886d5efb2e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg" podUID="2caf39c1-81ac-40c1-ade7-96886d5efb2e"
Jul 17 13:20:54.718312 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:54.718288    2565 scope.go:117] "RemoveContainer" containerID="cc3706c21c613d8617879f2340cb50820beba0a6774060d2a21b856b089e6b17"
Jul 17 13:20:54.718676 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:20:54.718482    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg_odh-ai-gateway-infra(2caf39c1-81ac-40c1-ade7-96886d5efb2e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg" podUID="2caf39c1-81ac-40c1-ade7-96886d5efb2e"
Jul 17 13:20:55.423587 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:20:55.423550    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:21:00.021171 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:00.021098    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:21:05.094440 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:05.094404    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:21:08.289715 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:08.289670    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:21:08.381253 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:08.381218    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:21:09.396758 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:09.396719    2565 scope.go:117] "RemoveContainer" containerID="cc3706c21c613d8617879f2340cb50820beba0a6774060d2a21b856b089e6b17"
Jul 17 13:21:09.810691 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:09.810668    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg" event={"ID":"2caf39c1-81ac-40c1-ade7-96886d5efb2e","Type":"ContainerStarted","Data":"08a5e7238038dd855a40009c76bb464fdb0eba42732c195f9a3e237ae77fc727"}
Jul 17 13:21:09.829556 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:09.829505    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg" podStartSLOduration=18.829488955 podStartE2EDuration="18.829488955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:21:09.827227246 +0000 UTC m=+3513.958611928" watchObservedRunningTime="2026-07-17 13:21:09.829488955 +0000 UTC m=+3513.960873638"
Jul 17 13:21:10.818485 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:10.818453    2565 generic.go:356] "Generic (PLEG): container finished" podID="2caf39c1-81ac-40c1-ade7-96886d5efb2e" containerID="08a5e7238038dd855a40009c76bb464fdb0eba42732c195f9a3e237ae77fc727" exitCode=6
Jul 17 13:21:10.818864 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:10.818525    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg" event={"ID":"2caf39c1-81ac-40c1-ade7-96886d5efb2e","Type":"ContainerDied","Data":"08a5e7238038dd855a40009c76bb464fdb0eba42732c195f9a3e237ae77fc727"}
Jul 17 13:21:10.818864 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:10.818582    2565 scope.go:117] "RemoveContainer" containerID="cc3706c21c613d8617879f2340cb50820beba0a6774060d2a21b856b089e6b17"
Jul 17 13:21:10.818864 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:10.818831    2565 scope.go:117] "RemoveContainer" containerID="08a5e7238038dd855a40009c76bb464fdb0eba42732c195f9a3e237ae77fc727"
Jul 17 13:21:10.819081 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:21:10.819059    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg_odh-ai-gateway-infra(2caf39c1-81ac-40c1-ade7-96886d5efb2e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg" podUID="2caf39c1-81ac-40c1-ade7-96886d5efb2e"
Jul 17 13:21:10.841092 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:10.841070    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg"]
Jul 17 13:21:11.971726 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:11.971697    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg"
Jul 17 13:21:12.073160 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:12.073132    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2caf39c1-81ac-40c1-ade7-96886d5efb2e-maas-api-service-ca\") pod \"2caf39c1-81ac-40c1-ade7-96886d5efb2e\" (UID: \"2caf39c1-81ac-40c1-ade7-96886d5efb2e\") "
Jul 17 13:21:12.073476 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:12.073456    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2caf39c1-81ac-40c1-ade7-96886d5efb2e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2caf39c1-81ac-40c1-ade7-96886d5efb2e" (UID: "2caf39c1-81ac-40c1-ade7-96886d5efb2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:21:12.174126 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:12.174063    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2caf39c1-81ac-40c1-ade7-96886d5efb2e-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:21:12.833248 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:12.833221    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg"
Jul 17 13:21:12.833415 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:12.833268    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg" event={"ID":"2caf39c1-81ac-40c1-ade7-96886d5efb2e","Type":"ContainerDied","Data":"e7d5c5247d56789706657d0e75d3fd3833515153c9c6a58db6d2cacdfc2b8095"}
Jul 17 13:21:12.833415 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:12.833343    2565 scope.go:117] "RemoveContainer" containerID="08a5e7238038dd855a40009c76bb464fdb0eba42732c195f9a3e237ae77fc727"
Jul 17 13:21:12.855775 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:12.855741    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg"]
Jul 17 13:21:12.863100 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:12.863078    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-r7kjg"]
Jul 17 13:21:14.403059 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:14.403027    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2caf39c1-81ac-40c1-ade7-96886d5efb2e" path="/var/lib/kubelet/pods/2caf39c1-81ac-40c1-ade7-96886d5efb2e/volumes"
Jul 17 13:21:29.960695 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:29.960668    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj"]
Jul 17 13:21:29.961113 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:29.961099    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2caf39c1-81ac-40c1-ade7-96886d5efb2e" containerName="revoke-keys"
Jul 17 13:21:29.961113 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:29.961114    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2caf39c1-81ac-40c1-ade7-96886d5efb2e" containerName="revoke-keys"
Jul 17 13:21:29.961190 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:29.961127    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2caf39c1-81ac-40c1-ade7-96886d5efb2e" containerName="revoke-keys"
Jul 17 13:21:29.961190 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:29.961133    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2caf39c1-81ac-40c1-ade7-96886d5efb2e" containerName="revoke-keys"
Jul 17 13:21:29.961253 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:29.961210    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2caf39c1-81ac-40c1-ade7-96886d5efb2e" containerName="revoke-keys"
Jul 17 13:21:29.961253 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:29.961218    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2caf39c1-81ac-40c1-ade7-96886d5efb2e" containerName="revoke-keys"
Jul 17 13:21:29.961253 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:29.961224    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2caf39c1-81ac-40c1-ade7-96886d5efb2e" containerName="revoke-keys"
Jul 17 13:21:29.964336 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:29.964321    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj"
Jul 17 13:21:29.967147 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:29.967129    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:21:29.973456 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:29.973423    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj"]
Jul 17 13:21:30.121656 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:30.121620    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3cff19b-9cfb-410f-9491-9caba92de654-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj\" (UID: \"d3cff19b-9cfb-410f-9491-9caba92de654\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj"
Jul 17 13:21:30.223151 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:30.223063    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3cff19b-9cfb-410f-9491-9caba92de654-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj\" (UID: \"d3cff19b-9cfb-410f-9491-9caba92de654\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj"
Jul 17 13:21:30.223671 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:30.223650    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3cff19b-9cfb-410f-9491-9caba92de654-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj\" (UID: \"d3cff19b-9cfb-410f-9491-9caba92de654\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj"
Jul 17 13:21:30.276208 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:30.276180    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj"
Jul 17 13:21:30.423712 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:30.423689    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj"]
Jul 17 13:21:30.424749 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:21:30.424727    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3cff19b_9cfb_410f_9491_9caba92de654.slice/crio-fafa3ecba8f1c582e89c34703c88b72030eed7c2ac4ad1d2cfcf309fe491327f WatchSource:0}: Error finding container fafa3ecba8f1c582e89c34703c88b72030eed7c2ac4ad1d2cfcf309fe491327f: Status 404 returned error can't find the container with id fafa3ecba8f1c582e89c34703c88b72030eed7c2ac4ad1d2cfcf309fe491327f
Jul 17 13:21:30.949430 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:30.949341    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj" event={"ID":"d3cff19b-9cfb-410f-9491-9caba92de654","Type":"ContainerStarted","Data":"b8a96a2d08ddb8c43b4a6b52b6bae0ff68adee15a16badde27c98116b162f807"}
Jul 17 13:21:30.949430 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:30.949398    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj" event={"ID":"d3cff19b-9cfb-410f-9491-9caba92de654","Type":"ContainerStarted","Data":"fafa3ecba8f1c582e89c34703c88b72030eed7c2ac4ad1d2cfcf309fe491327f"}
Jul 17 13:21:30.970295 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:30.970249    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj" podStartSLOduration=1.970234211 podStartE2EDuration="1.970234211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:21:30.968379671 +0000 UTC m=+3535.099764322" watchObservedRunningTime="2026-07-17 13:21:30.970234211 +0000 UTC m=+3535.101618866"
Jul 17 13:21:31.956344 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:31.956309    2565 generic.go:356] "Generic (PLEG): container finished" podID="d3cff19b-9cfb-410f-9491-9caba92de654" containerID="b8a96a2d08ddb8c43b4a6b52b6bae0ff68adee15a16badde27c98116b162f807" exitCode=6
Jul 17 13:21:31.956541 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:31.956395    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj" event={"ID":"d3cff19b-9cfb-410f-9491-9caba92de654","Type":"ContainerDied","Data":"b8a96a2d08ddb8c43b4a6b52b6bae0ff68adee15a16badde27c98116b162f807"}
Jul 17 13:21:31.956541 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:31.956512    2565 scope.go:117] "RemoveContainer" containerID="b8a96a2d08ddb8c43b4a6b52b6bae0ff68adee15a16badde27c98116b162f807"
Jul 17 13:21:32.964455 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:32.964380    2565 generic.go:356] "Generic (PLEG): container finished" podID="d3cff19b-9cfb-410f-9491-9caba92de654" containerID="57fd0729160bee433c1671e561244e354691cb810b7ef1eea1e494796c1407ef" exitCode=6
Jul 17 13:21:32.964827 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:32.964463    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj" event={"ID":"d3cff19b-9cfb-410f-9491-9caba92de654","Type":"ContainerDied","Data":"57fd0729160bee433c1671e561244e354691cb810b7ef1eea1e494796c1407ef"}
Jul 17 13:21:32.964827 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:32.964519    2565 scope.go:117] "RemoveContainer" containerID="b8a96a2d08ddb8c43b4a6b52b6bae0ff68adee15a16badde27c98116b162f807"
Jul 17 13:21:32.964827 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:32.964683    2565 scope.go:117] "RemoveContainer" containerID="57fd0729160bee433c1671e561244e354691cb810b7ef1eea1e494796c1407ef"
Jul 17 13:21:32.964964 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:21:32.964902    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj_odh-ai-gateway-infra(d3cff19b-9cfb-410f-9491-9caba92de654)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj" podUID="d3cff19b-9cfb-410f-9491-9caba92de654"
Jul 17 13:21:33.972401 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:33.972375    2565 scope.go:117] "RemoveContainer" containerID="57fd0729160bee433c1671e561244e354691cb810b7ef1eea1e494796c1407ef"
Jul 17 13:21:33.972846 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:21:33.972587    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj_odh-ai-gateway-infra(d3cff19b-9cfb-410f-9491-9caba92de654)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj" podUID="d3cff19b-9cfb-410f-9491-9caba92de654"
Jul 17 13:21:42.265267 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:42.265233    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:21:43.746701 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:43.746667    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:21:47.397003 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:47.396962    2565 scope.go:117] "RemoveContainer" containerID="57fd0729160bee433c1671e561244e354691cb810b7ef1eea1e494796c1407ef"
Jul 17 13:21:48.059064 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:48.059035    2565 generic.go:356] "Generic (PLEG): container finished" podID="d3cff19b-9cfb-410f-9491-9caba92de654" containerID="6abcfe2c953c04b5480c7ac04ff38911765d0dbdcd23b82ca50c7d0e0c097fd4" exitCode=6
Jul 17 13:21:48.059224 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:48.059111    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj" event={"ID":"d3cff19b-9cfb-410f-9491-9caba92de654","Type":"ContainerDied","Data":"6abcfe2c953c04b5480c7ac04ff38911765d0dbdcd23b82ca50c7d0e0c097fd4"}
Jul 17 13:21:48.059224 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:48.059172    2565 scope.go:117] "RemoveContainer" containerID="57fd0729160bee433c1671e561244e354691cb810b7ef1eea1e494796c1407ef"
Jul 17 13:21:48.059434 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:48.059422    2565 scope.go:117] "RemoveContainer" containerID="6abcfe2c953c04b5480c7ac04ff38911765d0dbdcd23b82ca50c7d0e0c097fd4"
Jul 17 13:21:48.059642 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:21:48.059623    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj_odh-ai-gateway-infra(d3cff19b-9cfb-410f-9491-9caba92de654)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj" podUID="d3cff19b-9cfb-410f-9491-9caba92de654"
Jul 17 13:21:48.161487 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:48.161453    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:21:48.262069 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:48.261974    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:21:49.094109 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:49.094078    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj"]
Jul 17 13:21:49.234355 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:49.234331    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj"
Jul 17 13:21:49.292009 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:49.291972    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3cff19b-9cfb-410f-9491-9caba92de654-maas-api-service-ca\") pod \"d3cff19b-9cfb-410f-9491-9caba92de654\" (UID: \"d3cff19b-9cfb-410f-9491-9caba92de654\") "
Jul 17 13:21:49.292327 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:49.292303    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3cff19b-9cfb-410f-9491-9caba92de654-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d3cff19b-9cfb-410f-9491-9caba92de654" (UID: "d3cff19b-9cfb-410f-9491-9caba92de654"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:21:49.392704 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:49.392647    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3cff19b-9cfb-410f-9491-9caba92de654-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:21:50.074337 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:50.074306    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj"
Jul 17 13:21:50.074509 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:50.074344    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj" event={"ID":"d3cff19b-9cfb-410f-9491-9caba92de654","Type":"ContainerDied","Data":"fafa3ecba8f1c582e89c34703c88b72030eed7c2ac4ad1d2cfcf309fe491327f"}
Jul 17 13:21:50.074509 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:50.074406    2565 scope.go:117] "RemoveContainer" containerID="6abcfe2c953c04b5480c7ac04ff38911765d0dbdcd23b82ca50c7d0e0c097fd4"
Jul 17 13:21:50.102747 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:50.102717    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj"]
Jul 17 13:21:50.108843 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:50.108821    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-sbgsj"]
Jul 17 13:21:50.403356 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:50.403282    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3cff19b-9cfb-410f-9491-9caba92de654" path="/var/lib/kubelet/pods/d3cff19b-9cfb-410f-9491-9caba92de654/volumes"
Jul 17 13:21:51.192782 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:51.192746    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n"]
Jul 17 13:21:51.193381 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:51.193365    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3cff19b-9cfb-410f-9491-9caba92de654" containerName="revoke-keys"
Jul 17 13:21:51.193434 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:51.193384    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3cff19b-9cfb-410f-9491-9caba92de654" containerName="revoke-keys"
Jul 17 13:21:51.193434 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:51.193399    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3cff19b-9cfb-410f-9491-9caba92de654" containerName="revoke-keys"
Jul 17 13:21:51.193434 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:51.193406    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3cff19b-9cfb-410f-9491-9caba92de654" containerName="revoke-keys"
Jul 17 13:21:51.193434 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:51.193428    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2caf39c1-81ac-40c1-ade7-96886d5efb2e" containerName="revoke-keys"
Jul 17 13:21:51.193557 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:51.193436    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2caf39c1-81ac-40c1-ade7-96886d5efb2e" containerName="revoke-keys"
Jul 17 13:21:51.193557 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:51.193445    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3cff19b-9cfb-410f-9491-9caba92de654" containerName="revoke-keys"
Jul 17 13:21:51.193557 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:51.193453    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3cff19b-9cfb-410f-9491-9caba92de654" containerName="revoke-keys"
Jul 17 13:21:51.193653 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:51.193563    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3cff19b-9cfb-410f-9491-9caba92de654" containerName="revoke-keys"
Jul 17 13:21:51.193653 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:51.193573    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3cff19b-9cfb-410f-9491-9caba92de654" containerName="revoke-keys"
Jul 17 13:21:51.193653 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:51.193583    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3cff19b-9cfb-410f-9491-9caba92de654" containerName="revoke-keys"
Jul 17 13:21:51.198174 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:51.198158    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n"
Jul 17 13:21:51.201461 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:51.201400    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:21:51.204213 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:51.204192    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n"]
Jul 17 13:21:51.307372 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:51.307339    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920d8c91-3d9a-44f7-808b-2536b2715191-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-s576n\" (UID: \"920d8c91-3d9a-44f7-808b-2536b2715191\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n"
Jul 17 13:21:51.407679 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:51.407647    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920d8c91-3d9a-44f7-808b-2536b2715191-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-s576n\" (UID: \"920d8c91-3d9a-44f7-808b-2536b2715191\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n"
Jul 17 13:21:51.408466 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:51.408438    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920d8c91-3d9a-44f7-808b-2536b2715191-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-s576n\" (UID: \"920d8c91-3d9a-44f7-808b-2536b2715191\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n"
Jul 17 13:21:51.513931 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:51.513867    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n"
Jul 17 13:21:51.650033 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:51.650011    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n"]
Jul 17 13:21:51.652628 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:21:51.652592    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920d8c91_3d9a_44f7_808b_2536b2715191.slice/crio-4a6c4d9145a4219624912f2d93ed9864d4fac08b75024f5f11a9bf94bd371b2c WatchSource:0}: Error finding container 4a6c4d9145a4219624912f2d93ed9864d4fac08b75024f5f11a9bf94bd371b2c: Status 404 returned error can't find the container with id 4a6c4d9145a4219624912f2d93ed9864d4fac08b75024f5f11a9bf94bd371b2c
Jul 17 13:21:52.089170 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:52.089087    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n" event={"ID":"920d8c91-3d9a-44f7-808b-2536b2715191","Type":"ContainerStarted","Data":"8105c9a8233a87261d9f48ddc58155d0490896cf07cdc94dd85718e8a33b531b"}
Jul 17 13:21:52.089170 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:52.089140    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n" event={"ID":"920d8c91-3d9a-44f7-808b-2536b2715191","Type":"ContainerStarted","Data":"4a6c4d9145a4219624912f2d93ed9864d4fac08b75024f5f11a9bf94bd371b2c"}
Jul 17 13:21:52.106658 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:52.106609    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n" podStartSLOduration=1.106594891 podStartE2EDuration="1.106594891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:21:52.105634062 +0000 UTC m=+3556.237018724" watchObservedRunningTime="2026-07-17 13:21:52.106594891 +0000 UTC m=+3556.237979549"
Jul 17 13:21:52.291198 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:52.291169    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:21:52.515544 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:52.515467    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:21:53.096412 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:53.096374    2565 generic.go:356] "Generic (PLEG): container finished" podID="920d8c91-3d9a-44f7-808b-2536b2715191" containerID="8105c9a8233a87261d9f48ddc58155d0490896cf07cdc94dd85718e8a33b531b" exitCode=6
Jul 17 13:21:53.096570 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:53.096421    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n" event={"ID":"920d8c91-3d9a-44f7-808b-2536b2715191","Type":"ContainerDied","Data":"8105c9a8233a87261d9f48ddc58155d0490896cf07cdc94dd85718e8a33b531b"}
Jul 17 13:21:53.096640 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:53.096628    2565 scope.go:117] "RemoveContainer" containerID="8105c9a8233a87261d9f48ddc58155d0490896cf07cdc94dd85718e8a33b531b"
Jul 17 13:21:53.605563 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:53.605531    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:21:53.716606 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:53.716544    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:21:53.897950 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:53.897920    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:21:54.104179 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:54.104149    2565 generic.go:356] "Generic (PLEG): container finished" podID="920d8c91-3d9a-44f7-808b-2536b2715191" containerID="9753166d02a514d813cab3b5732973589f2762bf90a2bd307c59728f9ac19e68" exitCode=6
Jul 17 13:21:54.104342 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:54.104242    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n" event={"ID":"920d8c91-3d9a-44f7-808b-2536b2715191","Type":"ContainerDied","Data":"9753166d02a514d813cab3b5732973589f2762bf90a2bd307c59728f9ac19e68"}
Jul 17 13:21:54.104342 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:54.104330    2565 scope.go:117] "RemoveContainer" containerID="8105c9a8233a87261d9f48ddc58155d0490896cf07cdc94dd85718e8a33b531b"
Jul 17 13:21:54.104575 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:54.104560    2565 scope.go:117] "RemoveContainer" containerID="9753166d02a514d813cab3b5732973589f2762bf90a2bd307c59728f9ac19e68"
Jul 17 13:21:54.104799 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:21:54.104777    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-s576n_odh-ai-gateway-infra(920d8c91-3d9a-44f7-808b-2536b2715191)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n" podUID="920d8c91-3d9a-44f7-808b-2536b2715191"
Jul 17 13:21:54.685844 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:54.685809    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:21:54.781855 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:54.781828    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:21:55.112641 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:55.112612    2565 scope.go:117] "RemoveContainer" containerID="9753166d02a514d813cab3b5732973589f2762bf90a2bd307c59728f9ac19e68"
Jul 17 13:21:55.112890 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:21:55.112854    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-s576n_odh-ai-gateway-infra(920d8c91-3d9a-44f7-808b-2536b2715191)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n" podUID="920d8c91-3d9a-44f7-808b-2536b2715191"
Jul 17 13:21:56.222048 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:21:56.222011    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:02.023597 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:02.023564    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:02.109406 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:02.109373    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:02.737585 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:02.737548    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:07.397112 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:07.397085    2565 scope.go:117] "RemoveContainer" containerID="9753166d02a514d813cab3b5732973589f2762bf90a2bd307c59728f9ac19e68"
Jul 17 13:22:08.193102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:08.193079    2565 generic.go:356] "Generic (PLEG): container finished" podID="920d8c91-3d9a-44f7-808b-2536b2715191" containerID="a7f27e007f480af71fe4f562c5e529489ab546243b7c026d7bccf8f3f651dfd5" exitCode=6
Jul 17 13:22:08.193230 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:08.193146    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n" event={"ID":"920d8c91-3d9a-44f7-808b-2536b2715191","Type":"ContainerDied","Data":"a7f27e007f480af71fe4f562c5e529489ab546243b7c026d7bccf8f3f651dfd5"}
Jul 17 13:22:08.193230 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:08.193202    2565 scope.go:117] "RemoveContainer" containerID="9753166d02a514d813cab3b5732973589f2762bf90a2bd307c59728f9ac19e68"
Jul 17 13:22:08.193495 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:08.193474    2565 scope.go:117] "RemoveContainer" containerID="a7f27e007f480af71fe4f562c5e529489ab546243b7c026d7bccf8f3f651dfd5"
Jul 17 13:22:08.193674 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:22:08.193656    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-s576n_odh-ai-gateway-infra(920d8c91-3d9a-44f7-808b-2536b2715191)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n" podUID="920d8c91-3d9a-44f7-808b-2536b2715191"
Jul 17 13:22:09.235014 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:09.234969    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n"]
Jul 17 13:22:09.381896 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:09.381873    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n"
Jul 17 13:22:09.403430 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:09.403403    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:09.455926 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:09.455900    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920d8c91-3d9a-44f7-808b-2536b2715191-maas-api-service-ca\") pod \"920d8c91-3d9a-44f7-808b-2536b2715191\" (UID: \"920d8c91-3d9a-44f7-808b-2536b2715191\") "
Jul 17 13:22:09.456226 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:09.456207    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920d8c91-3d9a-44f7-808b-2536b2715191-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "920d8c91-3d9a-44f7-808b-2536b2715191" (UID: "920d8c91-3d9a-44f7-808b-2536b2715191"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:22:09.530176 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:09.530119    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:09.557542 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:09.557522    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/920d8c91-3d9a-44f7-808b-2536b2715191-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:22:09.673494 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:09.673455    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:09.798920 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:09.798891    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:10.207746 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:10.207663    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n"
Jul 17 13:22:10.207898 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:10.207665    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n" event={"ID":"920d8c91-3d9a-44f7-808b-2536b2715191","Type":"ContainerDied","Data":"4a6c4d9145a4219624912f2d93ed9864d4fac08b75024f5f11a9bf94bd371b2c"}
Jul 17 13:22:10.207898 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:10.207792    2565 scope.go:117] "RemoveContainer" containerID="a7f27e007f480af71fe4f562c5e529489ab546243b7c026d7bccf8f3f651dfd5"
Jul 17 13:22:10.238388 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:10.238365    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n"]
Jul 17 13:22:10.243474 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:10.243454    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-s576n"]
Jul 17 13:22:10.402588 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:10.402563    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="920d8c91-3d9a-44f7-808b-2536b2715191" path="/var/lib/kubelet/pods/920d8c91-3d9a-44f7-808b-2536b2715191/volumes"
Jul 17 13:22:20.241945 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:20.241887    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:20.595875 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:20.595846    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:36.726600 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:36.726571    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 13:22:36.732251 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:36.732230    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 13:22:36.753825 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:36.753802    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 13:22:36.758342 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:36.758323    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 13:22:41.275726 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:41.275693    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb"]
Jul 17 13:22:41.276137 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:41.276124    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920d8c91-3d9a-44f7-808b-2536b2715191" containerName="revoke-keys"
Jul 17 13:22:41.276185 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:41.276139    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="920d8c91-3d9a-44f7-808b-2536b2715191" containerName="revoke-keys"
Jul 17 13:22:41.276220 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:41.276192    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920d8c91-3d9a-44f7-808b-2536b2715191" containerName="revoke-keys"
Jul 17 13:22:41.276220 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:41.276197    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="920d8c91-3d9a-44f7-808b-2536b2715191" containerName="revoke-keys"
Jul 17 13:22:41.276279 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:41.276263    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="920d8c91-3d9a-44f7-808b-2536b2715191" containerName="revoke-keys"
Jul 17 13:22:41.276279 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:41.276273    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="920d8c91-3d9a-44f7-808b-2536b2715191" containerName="revoke-keys"
Jul 17 13:22:41.280745 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:41.280730    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb"
Jul 17 13:22:41.283920 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:41.283903    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:22:41.290035 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:41.290013    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb"]
Jul 17 13:22:41.301405 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:41.301385    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf2de59-e4c9-46c1-82ca-fa490d3e169b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb\" (UID: \"dbf2de59-e4c9-46c1-82ca-fa490d3e169b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb"
Jul 17 13:22:41.402490 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:41.402441    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf2de59-e4c9-46c1-82ca-fa490d3e169b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb\" (UID: \"dbf2de59-e4c9-46c1-82ca-fa490d3e169b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb"
Jul 17 13:22:41.403017 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:41.402977    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf2de59-e4c9-46c1-82ca-fa490d3e169b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb\" (UID: \"dbf2de59-e4c9-46c1-82ca-fa490d3e169b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb"
Jul 17 13:22:41.591958 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:41.591907    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb"
Jul 17 13:22:41.629445 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:41.629414    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:41.731764 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:41.731724    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb"]
Jul 17 13:22:41.733201 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:22:41.733174    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf2de59_e4c9_46c1_82ca_fa490d3e169b.slice/crio-57a7011ebec3c6f58d5dbf123203b30d13c80fc4bc600d3fbabf208aaf99289e WatchSource:0}: Error finding container 57a7011ebec3c6f58d5dbf123203b30d13c80fc4bc600d3fbabf208aaf99289e: Status 404 returned error can't find the container with id 57a7011ebec3c6f58d5dbf123203b30d13c80fc4bc600d3fbabf208aaf99289e
Jul 17 13:22:42.400825 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:42.400796    2565 generic.go:356] "Generic (PLEG): container finished" podID="dbf2de59-e4c9-46c1-82ca-fa490d3e169b" containerID="3cb435a4bca560d642e79e164f98902e339e5b5ecb95396f5a0379520f5ad71b" exitCode=6
Jul 17 13:22:42.401237 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:42.401110    2565 scope.go:117] "RemoveContainer" containerID="3cb435a4bca560d642e79e164f98902e339e5b5ecb95396f5a0379520f5ad71b"
Jul 17 13:22:42.403378 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:42.403290    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb" event={"ID":"dbf2de59-e4c9-46c1-82ca-fa490d3e169b","Type":"ContainerDied","Data":"3cb435a4bca560d642e79e164f98902e339e5b5ecb95396f5a0379520f5ad71b"}
Jul 17 13:22:42.403378 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:42.403329    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb" event={"ID":"dbf2de59-e4c9-46c1-82ca-fa490d3e169b","Type":"ContainerStarted","Data":"57a7011ebec3c6f58d5dbf123203b30d13c80fc4bc600d3fbabf208aaf99289e"}
Jul 17 13:22:43.408973 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:43.408883    2565 generic.go:356] "Generic (PLEG): container finished" podID="dbf2de59-e4c9-46c1-82ca-fa490d3e169b" containerID="1f274b5b3c38f89a907bd64146e1fbd737b7857199cf02a0c5bba4ef87b17f92" exitCode=6
Jul 17 13:22:43.408973 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:43.408931    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb" event={"ID":"dbf2de59-e4c9-46c1-82ca-fa490d3e169b","Type":"ContainerDied","Data":"1f274b5b3c38f89a907bd64146e1fbd737b7857199cf02a0c5bba4ef87b17f92"}
Jul 17 13:22:43.409441 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:43.409001    2565 scope.go:117] "RemoveContainer" containerID="3cb435a4bca560d642e79e164f98902e339e5b5ecb95396f5a0379520f5ad71b"
Jul 17 13:22:43.409441 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:43.409128    2565 scope.go:117] "RemoveContainer" containerID="1f274b5b3c38f89a907bd64146e1fbd737b7857199cf02a0c5bba4ef87b17f92"
Jul 17 13:22:43.409441 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:22:43.409345    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb_odh-ai-gateway-infra(dbf2de59-e4c9-46c1-82ca-fa490d3e169b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb" podUID="dbf2de59-e4c9-46c1-82ca-fa490d3e169b"
Jul 17 13:22:44.415767 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:44.415745    2565 scope.go:117] "RemoveContainer" containerID="1f274b5b3c38f89a907bd64146e1fbd737b7857199cf02a0c5bba4ef87b17f92"
Jul 17 13:22:44.416233 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:22:44.415953    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb_odh-ai-gateway-infra(dbf2de59-e4c9-46c1-82ca-fa490d3e169b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb" podUID="dbf2de59-e4c9-46c1-82ca-fa490d3e169b"
Jul 17 13:22:48.378978 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:48.378929    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:48.480385 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:48.480341    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:48.748862 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:48.748773    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:49.291485 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:49.291444    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:49.486758 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:49.486730    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:52.383037 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:52.382976    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:52.482447 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:52.482420    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:52.641491 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:52.641420    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:52.853461 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:52.853431    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:53.195766 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:53.195733    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:53.295289 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:53.295231    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:56.846494 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:56.846457    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:57.240018 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:57.239927    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:57.797253 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:57.797218    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:58.049149 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:58.049071    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:22:58.397046 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:58.396968    2565 scope.go:117] "RemoveContainer" containerID="1f274b5b3c38f89a907bd64146e1fbd737b7857199cf02a0c5bba4ef87b17f92"
Jul 17 13:22:59.496910 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:59.496882    2565 generic.go:356] "Generic (PLEG): container finished" podID="dbf2de59-e4c9-46c1-82ca-fa490d3e169b" containerID="8aaed1ee272a2f31fcaa0be2f3160d4daadad9d63d47e44039145a0361ed35ca" exitCode=6
Jul 17 13:22:59.497311 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:59.496956    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb" event={"ID":"dbf2de59-e4c9-46c1-82ca-fa490d3e169b","Type":"ContainerDied","Data":"8aaed1ee272a2f31fcaa0be2f3160d4daadad9d63d47e44039145a0361ed35ca"}
Jul 17 13:22:59.497311 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:59.497032    2565 scope.go:117] "RemoveContainer" containerID="1f274b5b3c38f89a907bd64146e1fbd737b7857199cf02a0c5bba4ef87b17f92"
Jul 17 13:22:59.497311 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:22:59.497280    2565 scope.go:117] "RemoveContainer" containerID="8aaed1ee272a2f31fcaa0be2f3160d4daadad9d63d47e44039145a0361ed35ca"
Jul 17 13:22:59.497478 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:22:59.497457    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb_odh-ai-gateway-infra(dbf2de59-e4c9-46c1-82ca-fa490d3e169b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb" podUID="dbf2de59-e4c9-46c1-82ca-fa490d3e169b"
Jul 17 13:23:00.404265 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:00.404199    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:23:00.493623 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:00.493594    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:23:00.535108 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:00.535076    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb"]
Jul 17 13:23:00.684774 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:00.684750    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb"
Jul 17 13:23:00.702867 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:00.702831    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:23:00.773671 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:00.773642    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf2de59-e4c9-46c1-82ca-fa490d3e169b-maas-api-service-ca\") pod \"dbf2de59-e4c9-46c1-82ca-fa490d3e169b\" (UID: \"dbf2de59-e4c9-46c1-82ca-fa490d3e169b\") "
Jul 17 13:23:00.773946 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:00.773927    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbf2de59-e4c9-46c1-82ca-fa490d3e169b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dbf2de59-e4c9-46c1-82ca-fa490d3e169b" (UID: "dbf2de59-e4c9-46c1-82ca-fa490d3e169b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:23:00.874564 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:00.874542    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf2de59-e4c9-46c1-82ca-fa490d3e169b-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:23:01.075977 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:01.075934    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:23:01.475943 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:01.475869    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:23:01.512890 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:01.512862    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb"
Jul 17 13:23:01.512890 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:01.512870    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb" event={"ID":"dbf2de59-e4c9-46c1-82ca-fa490d3e169b","Type":"ContainerDied","Data":"57a7011ebec3c6f58d5dbf123203b30d13c80fc4bc600d3fbabf208aaf99289e"}
Jul 17 13:23:01.513132 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:01.512928    2565 scope.go:117] "RemoveContainer" containerID="8aaed1ee272a2f31fcaa0be2f3160d4daadad9d63d47e44039145a0361ed35ca"
Jul 17 13:23:01.539884 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:01.539858    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb"]
Jul 17 13:23:01.546279 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:01.546258    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-k8hsb"]
Jul 17 13:23:01.578209 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:01.578171    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:23:02.403225 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:02.403194    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbf2de59-e4c9-46c1-82ca-fa490d3e169b" path="/var/lib/kubelet/pods/dbf2de59-e4c9-46c1-82ca-fa490d3e169b/volumes"
Jul 17 13:23:04.941035 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:04.940999    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:23:08.717899 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:08.717870    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:23:12.033212 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:12.033163    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:23:12.091784 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:12.091739    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:23:14.008630 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:14.008595    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:23:14.108855 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:14.108819    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:23:14.209244 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:14.209196    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:23:16.204814 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:16.204784    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:23:18.786039 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:18.786007    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb"]
Jul 17 13:23:18.786448 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:18.786437    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbf2de59-e4c9-46c1-82ca-fa490d3e169b" containerName="revoke-keys"
Jul 17 13:23:18.786490 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:18.786449    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbf2de59-e4c9-46c1-82ca-fa490d3e169b" containerName="revoke-keys"
Jul 17 13:23:18.786490 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:18.786465    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbf2de59-e4c9-46c1-82ca-fa490d3e169b" containerName="revoke-keys"
Jul 17 13:23:18.786490 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:18.786473    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbf2de59-e4c9-46c1-82ca-fa490d3e169b" containerName="revoke-keys"
Jul 17 13:23:18.786490 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:18.786488    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbf2de59-e4c9-46c1-82ca-fa490d3e169b" containerName="revoke-keys"
Jul 17 13:23:18.786614 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:18.786493    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbf2de59-e4c9-46c1-82ca-fa490d3e169b" containerName="revoke-keys"
Jul 17 13:23:18.786614 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:18.786511    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920d8c91-3d9a-44f7-808b-2536b2715191" containerName="revoke-keys"
Jul 17 13:23:18.786614 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:18.786516    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="920d8c91-3d9a-44f7-808b-2536b2715191" containerName="revoke-keys"
Jul 17 13:23:18.786614 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:18.786580    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbf2de59-e4c9-46c1-82ca-fa490d3e169b" containerName="revoke-keys"
Jul 17 13:23:18.786614 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:18.786587    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="920d8c91-3d9a-44f7-808b-2536b2715191" containerName="revoke-keys"
Jul 17 13:23:18.786614 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:18.786593    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbf2de59-e4c9-46c1-82ca-fa490d3e169b" containerName="revoke-keys"
Jul 17 13:23:18.790905 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:18.790891    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb"
Jul 17 13:23:18.794935 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:18.794915    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:23:18.800030 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:18.800010    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb"]
Jul 17 13:23:18.922869 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:18.922841    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef2e142-e499-462c-8d3b-ac428460e8ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb\" (UID: \"9ef2e142-e499-462c-8d3b-ac428460e8ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb"
Jul 17 13:23:19.024033 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:19.024009    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef2e142-e499-462c-8d3b-ac428460e8ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb\" (UID: \"9ef2e142-e499-462c-8d3b-ac428460e8ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb"
Jul 17 13:23:19.024592 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:19.024574    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef2e142-e499-462c-8d3b-ac428460e8ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb\" (UID: \"9ef2e142-e499-462c-8d3b-ac428460e8ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb"
Jul 17 13:23:19.102509 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:19.102483    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb"
Jul 17 13:23:19.454664 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:19.454635    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb"]
Jul 17 13:23:19.455792 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:23:19.455765    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef2e142_e499_462c_8d3b_ac428460e8ed.slice/crio-bc1a29e520c8947af3a57f627674336a13a8dcf6776ef796c71d18c72c8cbf21 WatchSource:0}: Error finding container bc1a29e520c8947af3a57f627674336a13a8dcf6776ef796c71d18c72c8cbf21: Status 404 returned error can't find the container with id bc1a29e520c8947af3a57f627674336a13a8dcf6776ef796c71d18c72c8cbf21
Jul 17 13:23:19.615451 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:19.615415    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb" event={"ID":"9ef2e142-e499-462c-8d3b-ac428460e8ed","Type":"ContainerStarted","Data":"bc1a29e520c8947af3a57f627674336a13a8dcf6776ef796c71d18c72c8cbf21"}
Jul 17 13:23:20.622549 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:20.622517    2565 generic.go:356] "Generic (PLEG): container finished" podID="9ef2e142-e499-462c-8d3b-ac428460e8ed" containerID="ae86d33af49e7feae357a3f7afac2902405e6185f5dee88f97f39645295a0c85" exitCode=6
Jul 17 13:23:20.622957 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:20.622639    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb" event={"ID":"9ef2e142-e499-462c-8d3b-ac428460e8ed","Type":"ContainerDied","Data":"ae86d33af49e7feae357a3f7afac2902405e6185f5dee88f97f39645295a0c85"}
Jul 17 13:23:20.622957 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:20.622747    2565 scope.go:117] "RemoveContainer" containerID="ae86d33af49e7feae357a3f7afac2902405e6185f5dee88f97f39645295a0c85"
Jul 17 13:23:21.250550 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:21.250466    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:23:21.630708 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:21.630677    2565 generic.go:356] "Generic (PLEG): container finished" podID="9ef2e142-e499-462c-8d3b-ac428460e8ed" containerID="c4e9a51a446897df696cf4e851d0a3697617f24a89f491da50a6632e287000d1" exitCode=6
Jul 17 13:23:21.631101 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:21.630749    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb" event={"ID":"9ef2e142-e499-462c-8d3b-ac428460e8ed","Type":"ContainerDied","Data":"c4e9a51a446897df696cf4e851d0a3697617f24a89f491da50a6632e287000d1"}
Jul 17 13:23:21.631101 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:21.630812    2565 scope.go:117] "RemoveContainer" containerID="ae86d33af49e7feae357a3f7afac2902405e6185f5dee88f97f39645295a0c85"
Jul 17 13:23:21.631101 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:21.631049    2565 scope.go:117] "RemoveContainer" containerID="c4e9a51a446897df696cf4e851d0a3697617f24a89f491da50a6632e287000d1"
Jul 17 13:23:21.631280 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:23:21.631261    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb_odh-ai-gateway-infra(9ef2e142-e499-462c-8d3b-ac428460e8ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb" podUID="9ef2e142-e499-462c-8d3b-ac428460e8ed"
Jul 17 13:23:22.519244 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:22.519216    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:23:22.617452 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:22.617421    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:23:22.638067 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:22.638050    2565 scope.go:117] "RemoveContainer" containerID="c4e9a51a446897df696cf4e851d0a3697617f24a89f491da50a6632e287000d1"
Jul 17 13:23:22.638377 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:23:22.638218    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb_odh-ai-gateway-infra(9ef2e142-e499-462c-8d3b-ac428460e8ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb" podUID="9ef2e142-e499-462c-8d3b-ac428460e8ed"
Jul 17 13:23:37.396750 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:37.396726    2565 scope.go:117] "RemoveContainer" containerID="c4e9a51a446897df696cf4e851d0a3697617f24a89f491da50a6632e287000d1"
Jul 17 13:23:37.728306 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:37.728281    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb" event={"ID":"9ef2e142-e499-462c-8d3b-ac428460e8ed","Type":"ContainerStarted","Data":"8b78c810b77d574c8af0b590ba8f3d3b5cfad6dee1c3005de469d6a60e1dab8b"}
Jul 17 13:23:37.768949 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:37.768910    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb" podStartSLOduration=19.768897364 podStartE2EDuration="19.768897364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:23:37.768580569 +0000 UTC m=+3661.899965228" watchObservedRunningTime="2026-07-17 13:23:37.768897364 +0000 UTC m=+3661.900282029"
Jul 17 13:23:38.735666 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:38.735634    2565 generic.go:356] "Generic (PLEG): container finished" podID="9ef2e142-e499-462c-8d3b-ac428460e8ed" containerID="8b78c810b77d574c8af0b590ba8f3d3b5cfad6dee1c3005de469d6a60e1dab8b" exitCode=6
Jul 17 13:23:38.736067 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:38.735699    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb" event={"ID":"9ef2e142-e499-462c-8d3b-ac428460e8ed","Type":"ContainerDied","Data":"8b78c810b77d574c8af0b590ba8f3d3b5cfad6dee1c3005de469d6a60e1dab8b"}
Jul 17 13:23:38.736067 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:38.735757    2565 scope.go:117] "RemoveContainer" containerID="c4e9a51a446897df696cf4e851d0a3697617f24a89f491da50a6632e287000d1"
Jul 17 13:23:38.736207 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:38.736068    2565 scope.go:117] "RemoveContainer" containerID="8b78c810b77d574c8af0b590ba8f3d3b5cfad6dee1c3005de469d6a60e1dab8b"
Jul 17 13:23:38.736284 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:23:38.736265    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb_odh-ai-gateway-infra(9ef2e142-e499-462c-8d3b-ac428460e8ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb" podUID="9ef2e142-e499-462c-8d3b-ac428460e8ed"
Jul 17 13:23:38.777232 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:38.777200    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb"]
Jul 17 13:23:39.885482 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:39.885462    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb"
Jul 17 13:23:39.993491 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:39.993424    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef2e142-e499-462c-8d3b-ac428460e8ed-maas-api-service-ca\") pod \"9ef2e142-e499-462c-8d3b-ac428460e8ed\" (UID: \"9ef2e142-e499-462c-8d3b-ac428460e8ed\") "
Jul 17 13:23:39.993805 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:39.993782    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef2e142-e499-462c-8d3b-ac428460e8ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9ef2e142-e499-462c-8d3b-ac428460e8ed" (UID: "9ef2e142-e499-462c-8d3b-ac428460e8ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:23:40.094459 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:40.094431    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef2e142-e499-462c-8d3b-ac428460e8ed-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:23:40.750842 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:40.750768    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb" event={"ID":"9ef2e142-e499-462c-8d3b-ac428460e8ed","Type":"ContainerDied","Data":"bc1a29e520c8947af3a57f627674336a13a8dcf6776ef796c71d18c72c8cbf21"}
Jul 17 13:23:40.750842 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:40.750795    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb"
Jul 17 13:23:40.750842 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:40.750827    2565 scope.go:117] "RemoveContainer" containerID="8b78c810b77d574c8af0b590ba8f3d3b5cfad6dee1c3005de469d6a60e1dab8b"
Jul 17 13:23:40.770503 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:40.770480    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb"]
Jul 17 13:23:40.774255 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:40.774234    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-9mtzb"]
Jul 17 13:23:41.318564 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:41.318537    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4"]
Jul 17 13:23:41.318960 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:41.318946    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ef2e142-e499-462c-8d3b-ac428460e8ed" containerName="revoke-keys"
Jul 17 13:23:41.319039 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:41.318962    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ef2e142-e499-462c-8d3b-ac428460e8ed" containerName="revoke-keys"
Jul 17 13:23:41.319039 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:41.319014    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ef2e142-e499-462c-8d3b-ac428460e8ed" containerName="revoke-keys"
Jul 17 13:23:41.319039 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:41.319020    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ef2e142-e499-462c-8d3b-ac428460e8ed" containerName="revoke-keys"
Jul 17 13:23:41.319155 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:41.319081    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbf2de59-e4c9-46c1-82ca-fa490d3e169b" containerName="revoke-keys"
Jul 17 13:23:41.319155 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:41.319091    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ef2e142-e499-462c-8d3b-ac428460e8ed" containerName="revoke-keys"
Jul 17 13:23:41.319155 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:41.319099    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ef2e142-e499-462c-8d3b-ac428460e8ed" containerName="revoke-keys"
Jul 17 13:23:41.323411 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:41.323397    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4"
Jul 17 13:23:41.326226 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:41.326204    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:23:41.331664 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:41.331636    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4"]
Jul 17 13:23:41.404275 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:41.404245    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4822a9ee-c3e9-45dc-b14b-754849f7acad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4\" (UID: \"4822a9ee-c3e9-45dc-b14b-754849f7acad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4"
Jul 17 13:23:41.504694 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:41.504662    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4822a9ee-c3e9-45dc-b14b-754849f7acad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4\" (UID: \"4822a9ee-c3e9-45dc-b14b-754849f7acad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4"
Jul 17 13:23:41.505260 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:41.505239    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4822a9ee-c3e9-45dc-b14b-754849f7acad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4\" (UID: \"4822a9ee-c3e9-45dc-b14b-754849f7acad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4"
Jul 17 13:23:41.635941 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:41.635888    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4"
Jul 17 13:23:41.770953 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:41.770927    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4"]
Jul 17 13:23:41.772153 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:23:41.772127    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4822a9ee_c3e9_45dc_b14b_754849f7acad.slice/crio-2fe6eaf64cec32a1a594a61b903277b221fe65bc67f59df2f0cc63e8dad5844a WatchSource:0}: Error finding container 2fe6eaf64cec32a1a594a61b903277b221fe65bc67f59df2f0cc63e8dad5844a: Status 404 returned error can't find the container with id 2fe6eaf64cec32a1a594a61b903277b221fe65bc67f59df2f0cc63e8dad5844a
Jul 17 13:23:42.404295 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:42.404268    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ef2e142-e499-462c-8d3b-ac428460e8ed" path="/var/lib/kubelet/pods/9ef2e142-e499-462c-8d3b-ac428460e8ed/volumes"
Jul 17 13:23:42.765502 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:42.765434    2565 generic.go:356] "Generic (PLEG): container finished" podID="4822a9ee-c3e9-45dc-b14b-754849f7acad" containerID="f47863c509a450b20c6beff11cd27a9a0c3c9bfcdd77ce98a6fe0b5dff6b4954" exitCode=6
Jul 17 13:23:42.765502 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:42.765484    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4" event={"ID":"4822a9ee-c3e9-45dc-b14b-754849f7acad","Type":"ContainerDied","Data":"f47863c509a450b20c6beff11cd27a9a0c3c9bfcdd77ce98a6fe0b5dff6b4954"}
Jul 17 13:23:42.765742 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:42.765532    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4" event={"ID":"4822a9ee-c3e9-45dc-b14b-754849f7acad","Type":"ContainerStarted","Data":"2fe6eaf64cec32a1a594a61b903277b221fe65bc67f59df2f0cc63e8dad5844a"}
Jul 17 13:23:42.765802 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:42.765751    2565 scope.go:117] "RemoveContainer" containerID="f47863c509a450b20c6beff11cd27a9a0c3c9bfcdd77ce98a6fe0b5dff6b4954"
Jul 17 13:23:43.773247 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:43.773169    2565 generic.go:356] "Generic (PLEG): container finished" podID="4822a9ee-c3e9-45dc-b14b-754849f7acad" containerID="e7a5649027bfd59928cafd9d74586aa7e10ef1a45d232afab74d715e53a061db" exitCode=6
Jul 17 13:23:43.773647 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:43.773255    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4" event={"ID":"4822a9ee-c3e9-45dc-b14b-754849f7acad","Type":"ContainerDied","Data":"e7a5649027bfd59928cafd9d74586aa7e10ef1a45d232afab74d715e53a061db"}
Jul 17 13:23:43.773647 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:43.773314    2565 scope.go:117] "RemoveContainer" containerID="f47863c509a450b20c6beff11cd27a9a0c3c9bfcdd77ce98a6fe0b5dff6b4954"
Jul 17 13:23:43.773647 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:43.773488    2565 scope.go:117] "RemoveContainer" containerID="e7a5649027bfd59928cafd9d74586aa7e10ef1a45d232afab74d715e53a061db"
Jul 17 13:23:43.773770 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:23:43.773678    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4_odh-ai-gateway-infra(4822a9ee-c3e9-45dc-b14b-754849f7acad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4" podUID="4822a9ee-c3e9-45dc-b14b-754849f7acad"
Jul 17 13:23:44.780235 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:44.780210    2565 scope.go:117] "RemoveContainer" containerID="e7a5649027bfd59928cafd9d74586aa7e10ef1a45d232afab74d715e53a061db"
Jul 17 13:23:44.780624 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:23:44.780362    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4_odh-ai-gateway-infra(4822a9ee-c3e9-45dc-b14b-754849f7acad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4" podUID="4822a9ee-c3e9-45dc-b14b-754849f7acad"
Jul 17 13:23:58.397143 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:58.397071    2565 scope.go:117] "RemoveContainer" containerID="e7a5649027bfd59928cafd9d74586aa7e10ef1a45d232afab74d715e53a061db"
Jul 17 13:23:58.864893 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:58.864865    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4" event={"ID":"4822a9ee-c3e9-45dc-b14b-754849f7acad","Type":"ContainerStarted","Data":"5c82c2777b2687f5bad59f617176473bc4fb3dc3aa87c55f02abaa56972e699b"}
Jul 17 13:23:58.884667 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:58.884620    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4" podStartSLOduration=17.884607536 podStartE2EDuration="17.884607536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:23:58.88295503 +0000 UTC m=+3683.014339711" watchObservedRunningTime="2026-07-17 13:23:58.884607536 +0000 UTC m=+3683.015992194"
Jul 17 13:23:59.872831 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:59.872802    2565 generic.go:356] "Generic (PLEG): container finished" podID="4822a9ee-c3e9-45dc-b14b-754849f7acad" containerID="5c82c2777b2687f5bad59f617176473bc4fb3dc3aa87c55f02abaa56972e699b" exitCode=6
Jul 17 13:23:59.873219 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:59.872868    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4" event={"ID":"4822a9ee-c3e9-45dc-b14b-754849f7acad","Type":"ContainerDied","Data":"5c82c2777b2687f5bad59f617176473bc4fb3dc3aa87c55f02abaa56972e699b"}
Jul 17 13:23:59.873219 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:59.872923    2565 scope.go:117] "RemoveContainer" containerID="e7a5649027bfd59928cafd9d74586aa7e10ef1a45d232afab74d715e53a061db"
Jul 17 13:23:59.873332 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:59.873227    2565 scope.go:117] "RemoveContainer" containerID="5c82c2777b2687f5bad59f617176473bc4fb3dc3aa87c55f02abaa56972e699b"
Jul 17 13:23:59.873451 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:23:59.873433    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4_odh-ai-gateway-infra(4822a9ee-c3e9-45dc-b14b-754849f7acad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4" podUID="4822a9ee-c3e9-45dc-b14b-754849f7acad"
Jul 17 13:23:59.900453 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:23:59.900431    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4"]
Jul 17 13:24:01.022359 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:01.022339    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4"
Jul 17 13:24:01.062793 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:01.062761    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4822a9ee-c3e9-45dc-b14b-754849f7acad-maas-api-service-ca\") pod \"4822a9ee-c3e9-45dc-b14b-754849f7acad\" (UID: \"4822a9ee-c3e9-45dc-b14b-754849f7acad\") "
Jul 17 13:24:01.063093 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:01.063073    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4822a9ee-c3e9-45dc-b14b-754849f7acad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4822a9ee-c3e9-45dc-b14b-754849f7acad" (UID: "4822a9ee-c3e9-45dc-b14b-754849f7acad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:24:01.164159 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:01.164107    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4822a9ee-c3e9-45dc-b14b-754849f7acad-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:24:01.888291 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:01.888258    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4"
Jul 17 13:24:01.888487 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:01.888304    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4" event={"ID":"4822a9ee-c3e9-45dc-b14b-754849f7acad","Type":"ContainerDied","Data":"2fe6eaf64cec32a1a594a61b903277b221fe65bc67f59df2f0cc63e8dad5844a"}
Jul 17 13:24:01.888487 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:01.888395    2565 scope.go:117] "RemoveContainer" containerID="5c82c2777b2687f5bad59f617176473bc4fb3dc3aa87c55f02abaa56972e699b"
Jul 17 13:24:01.911994 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:01.911950    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4"]
Jul 17 13:24:01.918025 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:01.918002    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-vrtr4"]
Jul 17 13:24:02.403387 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:02.403359    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4822a9ee-c3e9-45dc-b14b-754849f7acad" path="/var/lib/kubelet/pods/4822a9ee-c3e9-45dc-b14b-754849f7acad/volumes"
Jul 17 13:24:15.892648 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:15.892605    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:24:18.871060 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:18.870978    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q"]
Jul 17 13:24:18.871476 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:18.871463    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4822a9ee-c3e9-45dc-b14b-754849f7acad" containerName="revoke-keys"
Jul 17 13:24:18.871520 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:18.871481    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4822a9ee-c3e9-45dc-b14b-754849f7acad" containerName="revoke-keys"
Jul 17 13:24:18.871520 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:18.871500    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4822a9ee-c3e9-45dc-b14b-754849f7acad" containerName="revoke-keys"
Jul 17 13:24:18.871520 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:18.871506    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4822a9ee-c3e9-45dc-b14b-754849f7acad" containerName="revoke-keys"
Jul 17 13:24:18.871520 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:18.871514    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ef2e142-e499-462c-8d3b-ac428460e8ed" containerName="revoke-keys"
Jul 17 13:24:18.871520 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:18.871520    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ef2e142-e499-462c-8d3b-ac428460e8ed" containerName="revoke-keys"
Jul 17 13:24:18.871670 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:18.871591    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ef2e142-e499-462c-8d3b-ac428460e8ed" containerName="revoke-keys"
Jul 17 13:24:18.871670 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:18.871600    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4822a9ee-c3e9-45dc-b14b-754849f7acad" containerName="revoke-keys"
Jul 17 13:24:18.871670 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:18.871609    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4822a9ee-c3e9-45dc-b14b-754849f7acad" containerName="revoke-keys"
Jul 17 13:24:18.875885 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:18.875870    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q"
Jul 17 13:24:18.883595 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:18.883577    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:24:18.910548 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:18.910523    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q"]
Jul 17 13:24:19.000482 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:19.000459    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e78e0e3-bdeb-4726-b1c2-3b759d30755f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q\" (UID: \"3e78e0e3-bdeb-4726-b1c2-3b759d30755f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q"
Jul 17 13:24:19.101558 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:19.101532    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e78e0e3-bdeb-4726-b1c2-3b759d30755f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q\" (UID: \"3e78e0e3-bdeb-4726-b1c2-3b759d30755f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q"
Jul 17 13:24:19.102215 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:19.102198    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e78e0e3-bdeb-4726-b1c2-3b759d30755f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q\" (UID: \"3e78e0e3-bdeb-4726-b1c2-3b759d30755f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q"
Jul 17 13:24:19.186098 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:19.186047    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q"
Jul 17 13:24:19.323255 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:19.323228    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q"]
Jul 17 13:24:19.324383 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:24:19.324361    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e78e0e3_bdeb_4726_b1c2_3b759d30755f.slice/crio-91bfb19c6ef0aaa55a7ac01091f43abdf2e82206d8f7f0825a25b104526152a3 WatchSource:0}: Error finding container 91bfb19c6ef0aaa55a7ac01091f43abdf2e82206d8f7f0825a25b104526152a3: Status 404 returned error can't find the container with id 91bfb19c6ef0aaa55a7ac01091f43abdf2e82206d8f7f0825a25b104526152a3
Jul 17 13:24:19.992010 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:19.991959    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q" event={"ID":"3e78e0e3-bdeb-4726-b1c2-3b759d30755f","Type":"ContainerStarted","Data":"ef26b4bb912f95d3d6d2c0614b11cf639d93ab5d21daa8cf095f9378fb0bd120"}
Jul 17 13:24:19.992420 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:19.992037    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q" event={"ID":"3e78e0e3-bdeb-4726-b1c2-3b759d30755f","Type":"ContainerStarted","Data":"91bfb19c6ef0aaa55a7ac01091f43abdf2e82206d8f7f0825a25b104526152a3"}
Jul 17 13:24:20.021310 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:20.021269    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q" podStartSLOduration=2.02125503 podStartE2EDuration="2.02125503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:24:20.020410297 +0000 UTC m=+3704.151794957" watchObservedRunningTime="2026-07-17 13:24:20.02125503 +0000 UTC m=+3704.152639684"
Jul 17 13:24:20.999604 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:20.999574    2565 generic.go:356] "Generic (PLEG): container finished" podID="3e78e0e3-bdeb-4726-b1c2-3b759d30755f" containerID="ef26b4bb912f95d3d6d2c0614b11cf639d93ab5d21daa8cf095f9378fb0bd120" exitCode=6
Jul 17 13:24:20.999971 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:20.999665    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q" event={"ID":"3e78e0e3-bdeb-4726-b1c2-3b759d30755f","Type":"ContainerDied","Data":"ef26b4bb912f95d3d6d2c0614b11cf639d93ab5d21daa8cf095f9378fb0bd120"}
Jul 17 13:24:20.999971 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:20.999806    2565 scope.go:117] "RemoveContainer" containerID="ef26b4bb912f95d3d6d2c0614b11cf639d93ab5d21daa8cf095f9378fb0bd120"
Jul 17 13:24:22.007864 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:22.007769    2565 generic.go:356] "Generic (PLEG): container finished" podID="3e78e0e3-bdeb-4726-b1c2-3b759d30755f" containerID="f3c7fbea9986bb7437c1b5535ab4943f6933ee8ac3eca072bbc601ad2384e337" exitCode=6
Jul 17 13:24:22.007864 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:22.007811    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q" event={"ID":"3e78e0e3-bdeb-4726-b1c2-3b759d30755f","Type":"ContainerDied","Data":"f3c7fbea9986bb7437c1b5535ab4943f6933ee8ac3eca072bbc601ad2384e337"}
Jul 17 13:24:22.008382 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:22.007880    2565 scope.go:117] "RemoveContainer" containerID="ef26b4bb912f95d3d6d2c0614b11cf639d93ab5d21daa8cf095f9378fb0bd120"
Jul 17 13:24:22.008382 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:22.008084    2565 scope.go:117] "RemoveContainer" containerID="f3c7fbea9986bb7437c1b5535ab4943f6933ee8ac3eca072bbc601ad2384e337"
Jul 17 13:24:22.008382 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:24:22.008293    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q_odh-ai-gateway-infra(3e78e0e3-bdeb-4726-b1c2-3b759d30755f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q" podUID="3e78e0e3-bdeb-4726-b1c2-3b759d30755f"
Jul 17 13:24:23.017664 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:23.017636    2565 scope.go:117] "RemoveContainer" containerID="f3c7fbea9986bb7437c1b5535ab4943f6933ee8ac3eca072bbc601ad2384e337"
Jul 17 13:24:23.018057 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:24:23.017815    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q_odh-ai-gateway-infra(3e78e0e3-bdeb-4726-b1c2-3b759d30755f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q" podUID="3e78e0e3-bdeb-4726-b1c2-3b759d30755f"
Jul 17 13:24:30.150448 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:30.150415    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq"]
Jul 17 13:24:30.150870 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:30.150856    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4822a9ee-c3e9-45dc-b14b-754849f7acad" containerName="revoke-keys"
Jul 17 13:24:30.150870 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:30.150871    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4822a9ee-c3e9-45dc-b14b-754849f7acad" containerName="revoke-keys"
Jul 17 13:24:30.150950 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:30.150942    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4822a9ee-c3e9-45dc-b14b-754849f7acad" containerName="revoke-keys"
Jul 17 13:24:30.155566 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:30.155552    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq"
Jul 17 13:24:30.161815 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:30.161792    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq"]
Jul 17 13:24:30.300292 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:30.300258    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22da81fe-6bfb-4565-9406-1962d4b15f58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq\" (UID: \"22da81fe-6bfb-4565-9406-1962d4b15f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq"
Jul 17 13:24:30.401084 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:30.400994    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22da81fe-6bfb-4565-9406-1962d4b15f58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq\" (UID: \"22da81fe-6bfb-4565-9406-1962d4b15f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq"
Jul 17 13:24:30.401762 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:30.401737    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22da81fe-6bfb-4565-9406-1962d4b15f58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq\" (UID: \"22da81fe-6bfb-4565-9406-1962d4b15f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq"
Jul 17 13:24:30.467690 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:30.467664    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq"
Jul 17 13:24:30.810000 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:30.809875    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq"]
Jul 17 13:24:30.812075 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:24:30.812028    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22da81fe_6bfb_4565_9406_1962d4b15f58.slice/crio-fabbb646c891e1832427ea7735f398805be4822734e4429b81b594a56238ec71 WatchSource:0}: Error finding container fabbb646c891e1832427ea7735f398805be4822734e4429b81b594a56238ec71: Status 404 returned error can't find the container with id fabbb646c891e1832427ea7735f398805be4822734e4429b81b594a56238ec71
Jul 17 13:24:31.065135 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:31.065067    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq" event={"ID":"22da81fe-6bfb-4565-9406-1962d4b15f58","Type":"ContainerStarted","Data":"fabbb646c891e1832427ea7735f398805be4822734e4429b81b594a56238ec71"}
Jul 17 13:24:32.073045 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:32.073009    2565 generic.go:356] "Generic (PLEG): container finished" podID="22da81fe-6bfb-4565-9406-1962d4b15f58" containerID="8bb74ce7f3ac55e9b16fc6ae888c078ddd927a8ca6c715896cb1b351b540a3e9" exitCode=6
Jul 17 13:24:32.073500 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:32.073052    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq" event={"ID":"22da81fe-6bfb-4565-9406-1962d4b15f58","Type":"ContainerDied","Data":"8bb74ce7f3ac55e9b16fc6ae888c078ddd927a8ca6c715896cb1b351b540a3e9"}
Jul 17 13:24:32.073500 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:32.073301    2565 scope.go:117] "RemoveContainer" containerID="8bb74ce7f3ac55e9b16fc6ae888c078ddd927a8ca6c715896cb1b351b540a3e9"
Jul 17 13:24:33.080593 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:33.080562    2565 generic.go:356] "Generic (PLEG): container finished" podID="22da81fe-6bfb-4565-9406-1962d4b15f58" containerID="8d17a718f0582390db6fa11b856fd8f534549b847e2452773519dd89051aa172" exitCode=6
Jul 17 13:24:33.080959 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:33.080611    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq" event={"ID":"22da81fe-6bfb-4565-9406-1962d4b15f58","Type":"ContainerDied","Data":"8d17a718f0582390db6fa11b856fd8f534549b847e2452773519dd89051aa172"}
Jul 17 13:24:33.080959 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:33.080657    2565 scope.go:117] "RemoveContainer" containerID="8bb74ce7f3ac55e9b16fc6ae888c078ddd927a8ca6c715896cb1b351b540a3e9"
Jul 17 13:24:33.080959 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:33.080805    2565 scope.go:117] "RemoveContainer" containerID="8d17a718f0582390db6fa11b856fd8f534549b847e2452773519dd89051aa172"
Jul 17 13:24:33.081113 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:24:33.081024    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq_odh-ai-gateway-infra(22da81fe-6bfb-4565-9406-1962d4b15f58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq" podUID="22da81fe-6bfb-4565-9406-1962d4b15f58"
Jul 17 13:24:34.087526 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:34.087504    2565 scope.go:117] "RemoveContainer" containerID="8d17a718f0582390db6fa11b856fd8f534549b847e2452773519dd89051aa172"
Jul 17 13:24:34.087874 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:24:34.087689    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq_odh-ai-gateway-infra(22da81fe-6bfb-4565-9406-1962d4b15f58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq" podUID="22da81fe-6bfb-4565-9406-1962d4b15f58"
Jul 17 13:24:37.397133 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:37.396649    2565 scope.go:117] "RemoveContainer" containerID="f3c7fbea9986bb7437c1b5535ab4943f6933ee8ac3eca072bbc601ad2384e337"
Jul 17 13:24:38.112863 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:38.112836    2565 generic.go:356] "Generic (PLEG): container finished" podID="3e78e0e3-bdeb-4726-b1c2-3b759d30755f" containerID="e3530f89e29753d102098b45973aba40d4e2eec071abbd6bb7fd973fbe29d495" exitCode=6
Jul 17 13:24:38.112972 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:38.112919    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q" event={"ID":"3e78e0e3-bdeb-4726-b1c2-3b759d30755f","Type":"ContainerDied","Data":"e3530f89e29753d102098b45973aba40d4e2eec071abbd6bb7fd973fbe29d495"}
Jul 17 13:24:38.113044 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:38.113011    2565 scope.go:117] "RemoveContainer" containerID="f3c7fbea9986bb7437c1b5535ab4943f6933ee8ac3eca072bbc601ad2384e337"
Jul 17 13:24:38.113363 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:38.113332    2565 scope.go:117] "RemoveContainer" containerID="e3530f89e29753d102098b45973aba40d4e2eec071abbd6bb7fd973fbe29d495"
Jul 17 13:24:38.113698 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:24:38.113593    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q_odh-ai-gateway-infra(3e78e0e3-bdeb-4726-b1c2-3b759d30755f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q" podUID="3e78e0e3-bdeb-4726-b1c2-3b759d30755f"
Jul 17 13:24:39.146255 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:39.146217    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q"]
Jul 17 13:24:39.290674 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:39.290648    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q"
Jul 17 13:24:39.375712 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:39.375685    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e78e0e3-bdeb-4726-b1c2-3b759d30755f-maas-api-service-ca\") pod \"3e78e0e3-bdeb-4726-b1c2-3b759d30755f\" (UID: \"3e78e0e3-bdeb-4726-b1c2-3b759d30755f\") "
Jul 17 13:24:39.376038 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:39.376015    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e78e0e3-bdeb-4726-b1c2-3b759d30755f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3e78e0e3-bdeb-4726-b1c2-3b759d30755f" (UID: "3e78e0e3-bdeb-4726-b1c2-3b759d30755f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:24:39.477213 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:39.477131    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e78e0e3-bdeb-4726-b1c2-3b759d30755f-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:24:39.599239 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:39.599189    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:24:40.129701 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:40.129669    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q"
Jul 17 13:24:40.129876 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:40.129702    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q" event={"ID":"3e78e0e3-bdeb-4726-b1c2-3b759d30755f","Type":"ContainerDied","Data":"91bfb19c6ef0aaa55a7ac01091f43abdf2e82206d8f7f0825a25b104526152a3"}
Jul 17 13:24:40.129876 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:40.129778    2565 scope.go:117] "RemoveContainer" containerID="e3530f89e29753d102098b45973aba40d4e2eec071abbd6bb7fd973fbe29d495"
Jul 17 13:24:40.157458 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:40.157434    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q"]
Jul 17 13:24:40.161024 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:40.161005    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-f4p2q"]
Jul 17 13:24:40.402934 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:40.402844    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e78e0e3-bdeb-4726-b1c2-3b759d30755f" path="/var/lib/kubelet/pods/3e78e0e3-bdeb-4726-b1c2-3b759d30755f/volumes"
Jul 17 13:24:40.513885 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:40.513848    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:24:40.612647 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:40.612607    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae44b3-nftxk"]
Jul 17 13:24:40.613252 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:40.613235    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e78e0e3-bdeb-4726-b1c2-3b759d30755f" containerName="revoke-keys"
Jul 17 13:24:40.613310 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:40.613254    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e78e0e3-bdeb-4726-b1c2-3b759d30755f" containerName="revoke-keys"
Jul 17 13:24:40.613310 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:40.613271    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e78e0e3-bdeb-4726-b1c2-3b759d30755f" containerName="revoke-keys"
Jul 17 13:24:40.613310 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:40.613277    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e78e0e3-bdeb-4726-b1c2-3b759d30755f" containerName="revoke-keys"
Jul 17 13:24:40.613310 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:40.613290    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e78e0e3-bdeb-4726-b1c2-3b759d30755f" containerName="revoke-keys"
Jul 17 13:24:40.613310 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:40.613296    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e78e0e3-bdeb-4726-b1c2-3b759d30755f" containerName="revoke-keys"
Jul 17 13:24:40.613480 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:40.613367    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e78e0e3-bdeb-4726-b1c2-3b759d30755f" containerName="revoke-keys"
Jul 17 13:24:40.613480 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:40.613380    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e78e0e3-bdeb-4726-b1c2-3b759d30755f" containerName="revoke-keys"
Jul 17 13:24:40.618060 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:40.618041    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae44b3-nftxk"
Jul 17 13:24:40.626264 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:40.626067    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae44b3-nftxk"]
Jul 17 13:24:40.686922 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:40.686843    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4febd3-cd03-4771-a48b-85c61b3e5b5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ae44b3-nftxk\" (UID: \"de4febd3-cd03-4771-a48b-85c61b3e5b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae44b3-nftxk"
Jul 17 13:24:40.788204 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:40.788170    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4febd3-cd03-4771-a48b-85c61b3e5b5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ae44b3-nftxk\" (UID: \"de4febd3-cd03-4771-a48b-85c61b3e5b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae44b3-nftxk"
Jul 17 13:24:40.788765 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:40.788746    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4febd3-cd03-4771-a48b-85c61b3e5b5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ae44b3-nftxk\" (UID: \"de4febd3-cd03-4771-a48b-85c61b3e5b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae44b3-nftxk"
Jul 17 13:24:40.930916 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:40.930885    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae44b3-nftxk"
Jul 17 13:24:41.113243 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:41.113209    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae44b3-nftxk"]
Jul 17 13:24:41.114617 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:24:41.114589    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde4febd3_cd03_4771_a48b_85c61b3e5b5b.slice/crio-460a7281ab92c62ce7c092ba5cebb089659b13220fd19ebafb582dc44bc0a798 WatchSource:0}: Error finding container 460a7281ab92c62ce7c092ba5cebb089659b13220fd19ebafb582dc44bc0a798: Status 404 returned error can't find the container with id 460a7281ab92c62ce7c092ba5cebb089659b13220fd19ebafb582dc44bc0a798
Jul 17 13:24:41.139134 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:41.139102    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae44b3-nftxk" event={"ID":"de4febd3-cd03-4771-a48b-85c61b3e5b5b","Type":"ContainerStarted","Data":"460a7281ab92c62ce7c092ba5cebb089659b13220fd19ebafb582dc44bc0a798"}
Jul 17 13:24:41.342414 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:41.342385    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:24:41.442462 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:41.442437    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:24:42.147925 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:42.147852    2565 generic.go:356] "Generic (PLEG): container finished" podID="de4febd3-cd03-4771-a48b-85c61b3e5b5b" containerID="f3b6c53f4df5bfbea74ec5f8845f5a2d9a0fe7d1147230f7c9faaeb2fd405b2c" exitCode=0
Jul 17 13:24:42.148091 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:42.147942    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae44b3-nftxk" event={"ID":"de4febd3-cd03-4771-a48b-85c61b3e5b5b","Type":"ContainerDied","Data":"f3b6c53f4df5bfbea74ec5f8845f5a2d9a0fe7d1147230f7c9faaeb2fd405b2c"}
Jul 17 13:24:43.293119 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:43.293096    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae44b3-nftxk"
Jul 17 13:24:43.310610 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:43.310547    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4febd3-cd03-4771-a48b-85c61b3e5b5b-maas-api-service-ca\") pod \"de4febd3-cd03-4771-a48b-85c61b3e5b5b\" (UID: \"de4febd3-cd03-4771-a48b-85c61b3e5b5b\") "
Jul 17 13:24:43.314865 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:43.311171    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de4febd3-cd03-4771-a48b-85c61b3e5b5b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "de4febd3-cd03-4771-a48b-85c61b3e5b5b" (UID: "de4febd3-cd03-4771-a48b-85c61b3e5b5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:24:43.314865 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:43.312100    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4febd3-cd03-4771-a48b-85c61b3e5b5b-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:24:44.161549 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:44.161517    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae44b3-nftxk"
Jul 17 13:24:44.161696 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:44.161550    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ae44b3-nftxk" event={"ID":"de4febd3-cd03-4771-a48b-85c61b3e5b5b","Type":"ContainerDied","Data":"460a7281ab92c62ce7c092ba5cebb089659b13220fd19ebafb582dc44bc0a798"}
Jul 17 13:24:44.161696 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:44.161602    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="460a7281ab92c62ce7c092ba5cebb089659b13220fd19ebafb582dc44bc0a798"
Jul 17 13:24:46.400320 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:46.400293    2565 scope.go:117] "RemoveContainer" containerID="8d17a718f0582390db6fa11b856fd8f534549b847e2452773519dd89051aa172"
Jul 17 13:24:47.180783 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:47.180750    2565 generic.go:356] "Generic (PLEG): container finished" podID="22da81fe-6bfb-4565-9406-1962d4b15f58" containerID="2ece498af023708904fbfe927208726ee8f9f56d4830226e1b18ab14dae3ee83" exitCode=6
Jul 17 13:24:47.180966 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:47.180827    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq" event={"ID":"22da81fe-6bfb-4565-9406-1962d4b15f58","Type":"ContainerDied","Data":"2ece498af023708904fbfe927208726ee8f9f56d4830226e1b18ab14dae3ee83"}
Jul 17 13:24:47.180966 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:47.180904    2565 scope.go:117] "RemoveContainer" containerID="8d17a718f0582390db6fa11b856fd8f534549b847e2452773519dd89051aa172"
Jul 17 13:24:47.181291 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:47.181275    2565 scope.go:117] "RemoveContainer" containerID="2ece498af023708904fbfe927208726ee8f9f56d4830226e1b18ab14dae3ee83"
Jul 17 13:24:47.181502 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:24:47.181483    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq_odh-ai-gateway-infra(22da81fe-6bfb-4565-9406-1962d4b15f58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq" podUID="22da81fe-6bfb-4565-9406-1962d4b15f58"
Jul 17 13:24:48.221625 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:48.221595    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq"]
Jul 17 13:24:48.366635 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:48.366614    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq"
Jul 17 13:24:48.458416 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:48.458386    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22da81fe-6bfb-4565-9406-1962d4b15f58-maas-api-service-ca\") pod \"22da81fe-6bfb-4565-9406-1962d4b15f58\" (UID: \"22da81fe-6bfb-4565-9406-1962d4b15f58\") "
Jul 17 13:24:48.458795 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:48.458771    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22da81fe-6bfb-4565-9406-1962d4b15f58-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "22da81fe-6bfb-4565-9406-1962d4b15f58" (UID: "22da81fe-6bfb-4565-9406-1962d4b15f58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:24:48.559592 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:48.559559    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22da81fe-6bfb-4565-9406-1962d4b15f58-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:24:49.194568 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:49.194539    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq"
Jul 17 13:24:49.194776 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:49.194590    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq" event={"ID":"22da81fe-6bfb-4565-9406-1962d4b15f58","Type":"ContainerDied","Data":"fabbb646c891e1832427ea7735f398805be4822734e4429b81b594a56238ec71"}
Jul 17 13:24:49.194776 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:49.194692    2565 scope.go:117] "RemoveContainer" containerID="2ece498af023708904fbfe927208726ee8f9f56d4830226e1b18ab14dae3ee83"
Jul 17 13:24:49.219548 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:49.219508    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq"]
Jul 17 13:24:49.222990 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:49.222965    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-hkdcq"]
Jul 17 13:24:50.404458 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:50.404428    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22da81fe-6bfb-4565-9406-1962d4b15f58" path="/var/lib/kubelet/pods/22da81fe-6bfb-4565-9406-1962d4b15f58/volumes"
Jul 17 13:24:51.282455 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:51.282420    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:24:55.751282 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:24:55.751252    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:02.691173 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:02.691139    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:02.815896 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:02.815866    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:06.478357 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:06.478323    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:06.695309 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:06.695275    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:07.468862 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:07.468838    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:07.568316 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:07.568261    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:17.292943 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:17.292902    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:20.191778 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:20.191740    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9"]
Jul 17 13:25:20.192201 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:20.192188    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22da81fe-6bfb-4565-9406-1962d4b15f58" containerName="revoke-keys"
Jul 17 13:25:20.192201 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:20.192202    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="22da81fe-6bfb-4565-9406-1962d4b15f58" containerName="revoke-keys"
Jul 17 13:25:20.192275 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:20.192209    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de4febd3-cd03-4771-a48b-85c61b3e5b5b" containerName="revoke-keys"
Jul 17 13:25:20.192275 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:20.192215    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="de4febd3-cd03-4771-a48b-85c61b3e5b5b" containerName="revoke-keys"
Jul 17 13:25:20.192275 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:20.192221    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22da81fe-6bfb-4565-9406-1962d4b15f58" containerName="revoke-keys"
Jul 17 13:25:20.192275 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:20.192226    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="22da81fe-6bfb-4565-9406-1962d4b15f58" containerName="revoke-keys"
Jul 17 13:25:20.192401 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:20.192299    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e78e0e3-bdeb-4726-b1c2-3b759d30755f" containerName="revoke-keys"
Jul 17 13:25:20.192401 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:20.192308    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="22da81fe-6bfb-4565-9406-1962d4b15f58" containerName="revoke-keys"
Jul 17 13:25:20.192401 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:20.192315    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="de4febd3-cd03-4771-a48b-85c61b3e5b5b" containerName="revoke-keys"
Jul 17 13:25:20.192401 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:20.192321    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="22da81fe-6bfb-4565-9406-1962d4b15f58" containerName="revoke-keys"
Jul 17 13:25:20.192401 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:20.192328    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="22da81fe-6bfb-4565-9406-1962d4b15f58" containerName="revoke-keys"
Jul 17 13:25:20.194955 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:20.194940    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9"
Jul 17 13:25:20.197879 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:20.197854    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:25:20.203117 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:20.203088    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9"]
Jul 17 13:25:20.352745 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:20.352702    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/297acd52-45fb-4836-8eb8-429d9e947b14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9\" (UID: \"297acd52-45fb-4836-8eb8-429d9e947b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9"
Jul 17 13:25:20.454452 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:20.454363    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/297acd52-45fb-4836-8eb8-429d9e947b14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9\" (UID: \"297acd52-45fb-4836-8eb8-429d9e947b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9"
Jul 17 13:25:20.454954 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:20.454937    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/297acd52-45fb-4836-8eb8-429d9e947b14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9\" (UID: \"297acd52-45fb-4836-8eb8-429d9e947b14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9"
Jul 17 13:25:20.506835 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:20.506801    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9"
Jul 17 13:25:20.653220 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:20.653191    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9"]
Jul 17 13:25:20.654748 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:25:20.654720    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297acd52_45fb_4836_8eb8_429d9e947b14.slice/crio-3d4919a1708b62a97c16cf85a9cb8fb39a7e0a5f6249ac301375e6f3c15ff952 WatchSource:0}: Error finding container 3d4919a1708b62a97c16cf85a9cb8fb39a7e0a5f6249ac301375e6f3c15ff952: Status 404 returned error can't find the container with id 3d4919a1708b62a97c16cf85a9cb8fb39a7e0a5f6249ac301375e6f3c15ff952
Jul 17 13:25:21.389655 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:21.389623    2565 generic.go:356] "Generic (PLEG): container finished" podID="297acd52-45fb-4836-8eb8-429d9e947b14" containerID="b97daa3b8fc76b7e62cd82a0e2a0c560d5d8d6894f8d20f68f10f6358e422b4f" exitCode=6
Jul 17 13:25:21.390099 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:21.389709    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9" event={"ID":"297acd52-45fb-4836-8eb8-429d9e947b14","Type":"ContainerDied","Data":"b97daa3b8fc76b7e62cd82a0e2a0c560d5d8d6894f8d20f68f10f6358e422b4f"}
Jul 17 13:25:21.390099 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:21.389762    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9" event={"ID":"297acd52-45fb-4836-8eb8-429d9e947b14","Type":"ContainerStarted","Data":"3d4919a1708b62a97c16cf85a9cb8fb39a7e0a5f6249ac301375e6f3c15ff952"}
Jul 17 13:25:21.390099 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:21.389873    2565 scope.go:117] "RemoveContainer" containerID="b97daa3b8fc76b7e62cd82a0e2a0c560d5d8d6894f8d20f68f10f6358e422b4f"
Jul 17 13:25:21.915780 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:21.915625    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:22.397585 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:22.397552    2565 generic.go:356] "Generic (PLEG): container finished" podID="297acd52-45fb-4836-8eb8-429d9e947b14" containerID="d8c0b503eaad78ca0fc573769bd26206f015b3490ce1cfd847a2175714979bd3" exitCode=6
Jul 17 13:25:22.398086 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:22.397975    2565 scope.go:117] "RemoveContainer" containerID="d8c0b503eaad78ca0fc573769bd26206f015b3490ce1cfd847a2175714979bd3"
Jul 17 13:25:22.398273 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:25:22.398251    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9_odh-ai-gateway-infra(297acd52-45fb-4836-8eb8-429d9e947b14)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9" podUID="297acd52-45fb-4836-8eb8-429d9e947b14"
Jul 17 13:25:22.403469 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:22.403431    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9" event={"ID":"297acd52-45fb-4836-8eb8-429d9e947b14","Type":"ContainerDied","Data":"d8c0b503eaad78ca0fc573769bd26206f015b3490ce1cfd847a2175714979bd3"}
Jul 17 13:25:22.403632 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:22.403499    2565 scope.go:117] "RemoveContainer" containerID="b97daa3b8fc76b7e62cd82a0e2a0c560d5d8d6894f8d20f68f10f6358e422b4f"
Jul 17 13:25:23.405763 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:23.405735    2565 scope.go:117] "RemoveContainer" containerID="d8c0b503eaad78ca0fc573769bd26206f015b3490ce1cfd847a2175714979bd3"
Jul 17 13:25:23.406163 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:25:23.405905    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9_odh-ai-gateway-infra(297acd52-45fb-4836-8eb8-429d9e947b14)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9" podUID="297acd52-45fb-4836-8eb8-429d9e947b14"
Jul 17 13:25:27.910441 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:27.910359    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:28.008708 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:28.008669    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:33.018704 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:33.018673    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:33.103753 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:33.103714    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:33.232642 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:33.232606    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:33.658243 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:33.658212    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:33.863143 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:33.863113    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:37.179321 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:37.179267    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:37.281394 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:37.281348    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:37.375388 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:37.375265    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:37.397289 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:37.397261    2565 scope.go:117] "RemoveContainer" containerID="d8c0b503eaad78ca0fc573769bd26206f015b3490ce1cfd847a2175714979bd3"
Jul 17 13:25:37.977302 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:37.977265    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:38.064593 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:38.064497    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:38.108949 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:25:38.108914    2565 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297acd52_45fb_4836_8eb8_429d9e947b14.slice/crio-166d6ffdcb1a8db5c150814a5d26964f7ccd9a8e0b7976af8e00d0898ed0402e.scope\": RecentStats: unable to find data in memory cache]"
Jul 17 13:25:38.163151 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:38.163120    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:38.499715 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:38.499628    2565 generic.go:356] "Generic (PLEG): container finished" podID="297acd52-45fb-4836-8eb8-429d9e947b14" containerID="166d6ffdcb1a8db5c150814a5d26964f7ccd9a8e0b7976af8e00d0898ed0402e" exitCode=6
Jul 17 13:25:38.499715 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:38.499702    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9" event={"ID":"297acd52-45fb-4836-8eb8-429d9e947b14","Type":"ContainerDied","Data":"166d6ffdcb1a8db5c150814a5d26964f7ccd9a8e0b7976af8e00d0898ed0402e"}
Jul 17 13:25:38.500185 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:38.499761    2565 scope.go:117] "RemoveContainer" containerID="d8c0b503eaad78ca0fc573769bd26206f015b3490ce1cfd847a2175714979bd3"
Jul 17 13:25:38.500185 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:38.500098    2565 scope.go:117] "RemoveContainer" containerID="166d6ffdcb1a8db5c150814a5d26964f7ccd9a8e0b7976af8e00d0898ed0402e"
Jul 17 13:25:38.500330 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:25:38.500312    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9_odh-ai-gateway-infra(297acd52-45fb-4836-8eb8-429d9e947b14)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9" podUID="297acd52-45fb-4836-8eb8-429d9e947b14"
Jul 17 13:25:39.534013 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:39.533951    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9"]
Jul 17 13:25:39.681317 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:39.681292    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9"
Jul 17 13:25:39.737830 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:39.737796    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/297acd52-45fb-4836-8eb8-429d9e947b14-maas-api-service-ca\") pod \"297acd52-45fb-4836-8eb8-429d9e947b14\" (UID: \"297acd52-45fb-4836-8eb8-429d9e947b14\") "
Jul 17 13:25:39.738195 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:39.738169    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/297acd52-45fb-4836-8eb8-429d9e947b14-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "297acd52-45fb-4836-8eb8-429d9e947b14" (UID: "297acd52-45fb-4836-8eb8-429d9e947b14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:25:39.838850 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:39.838810    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/297acd52-45fb-4836-8eb8-429d9e947b14-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:25:40.515388 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:40.515362    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9"
Jul 17 13:25:40.515600 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:40.515360    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9" event={"ID":"297acd52-45fb-4836-8eb8-429d9e947b14","Type":"ContainerDied","Data":"3d4919a1708b62a97c16cf85a9cb8fb39a7e0a5f6249ac301375e6f3c15ff952"}
Jul 17 13:25:40.515600 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:40.515505    2565 scope.go:117] "RemoveContainer" containerID="166d6ffdcb1a8db5c150814a5d26964f7ccd9a8e0b7976af8e00d0898ed0402e"
Jul 17 13:25:40.533247 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:40.533220    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9"]
Jul 17 13:25:40.535776 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:40.535754    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-m5tc9"]
Jul 17 13:25:41.586647 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:41.586614    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:42.067978 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:42.067949    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:42.393969 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:42.393880    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:42.403611 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:42.403575    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="297acd52-45fb-4836-8eb8-429d9e947b14" path="/var/lib/kubelet/pods/297acd52-45fb-4836-8eb8-429d9e947b14/volumes"
Jul 17 13:25:42.453564 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:42.453530    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:45.301168 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:45.301132    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:45.401089 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:45.401056    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:45.976562 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:45.976529    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:46.054030 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:46.053944    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:46.157960 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:46.157919    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:49.575003 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:49.574939    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:57.595843 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:57.595806    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771703-b4fbf"]
Jul 17 13:25:57.596266 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:57.596256    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="297acd52-45fb-4836-8eb8-429d9e947b14" containerName="revoke-keys"
Jul 17 13:25:57.596311 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:57.596268    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="297acd52-45fb-4836-8eb8-429d9e947b14" containerName="revoke-keys"
Jul 17 13:25:57.596311 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:57.596276    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22da81fe-6bfb-4565-9406-1962d4b15f58" containerName="revoke-keys"
Jul 17 13:25:57.596311 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:57.596282    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="22da81fe-6bfb-4565-9406-1962d4b15f58" containerName="revoke-keys"
Jul 17 13:25:57.596406 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:57.596312    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="297acd52-45fb-4836-8eb8-429d9e947b14" containerName="revoke-keys"
Jul 17 13:25:57.596406 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:57.596318    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="297acd52-45fb-4836-8eb8-429d9e947b14" containerName="revoke-keys"
Jul 17 13:25:57.596406 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:57.596378    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="297acd52-45fb-4836-8eb8-429d9e947b14" containerName="revoke-keys"
Jul 17 13:25:57.596406 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:57.596385    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="297acd52-45fb-4836-8eb8-429d9e947b14" containerName="revoke-keys"
Jul 17 13:25:57.599569 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:57.599552    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771703-b4fbf"
Jul 17 13:25:57.602443 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:57.602422    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:25:57.609037 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:57.609011    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771703-b4fbf"]
Jul 17 13:25:57.709875 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:57.709819    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88ec52a1-dbc0-4c2e-9db0-e83c0694fe9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-771703-b4fbf\" (UID: \"88ec52a1-dbc0-4c2e-9db0-e83c0694fe9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771703-b4fbf"
Jul 17 13:25:57.810371 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:57.810325    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88ec52a1-dbc0-4c2e-9db0-e83c0694fe9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-771703-b4fbf\" (UID: \"88ec52a1-dbc0-4c2e-9db0-e83c0694fe9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771703-b4fbf"
Jul 17 13:25:57.811076 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:57.811051    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88ec52a1-dbc0-4c2e-9db0-e83c0694fe9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-771703-b4fbf\" (UID: \"88ec52a1-dbc0-4c2e-9db0-e83c0694fe9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771703-b4fbf"
Jul 17 13:25:57.912652 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:57.912560    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771703-b4fbf"
Jul 17 13:25:58.062393 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:58.062364    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771703-b4fbf"]
Jul 17 13:25:58.063228 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:25:58.063201    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ec52a1_dbc0_4c2e_9db0_e83c0694fe9a.slice/crio-adb52f20d766e4ae59e0076453d1450d2242b74af1ed6c72b9800f5092a7dfbc WatchSource:0}: Error finding container adb52f20d766e4ae59e0076453d1450d2242b74af1ed6c72b9800f5092a7dfbc: Status 404 returned error can't find the container with id adb52f20d766e4ae59e0076453d1450d2242b74af1ed6c72b9800f5092a7dfbc
Jul 17 13:25:58.325903 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:58.325862    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:58.430232 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:58.430189    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:25:58.629860 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:58.629827    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771703-b4fbf" event={"ID":"88ec52a1-dbc0-4c2e-9db0-e83c0694fe9a","Type":"ContainerStarted","Data":"52018c559745a085861a4c1e90e6ce850da4b69c000425e40da37abc6906db01"}
Jul 17 13:25:58.630370 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:58.629878    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771703-b4fbf" event={"ID":"88ec52a1-dbc0-4c2e-9db0-e83c0694fe9a","Type":"ContainerStarted","Data":"adb52f20d766e4ae59e0076453d1450d2242b74af1ed6c72b9800f5092a7dfbc"}
Jul 17 13:25:58.658293 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:58.658236    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771703-b4fbf" podStartSLOduration=1.658218292 podStartE2EDuration="1.658218292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:25:58.654488161 +0000 UTC m=+3802.785872823" watchObservedRunningTime="2026-07-17 13:25:58.658218292 +0000 UTC m=+3802.789602953"
Jul 17 13:25:59.637408 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:59.637366    2565 generic.go:356] "Generic (PLEG): container finished" podID="88ec52a1-dbc0-4c2e-9db0-e83c0694fe9a" containerID="52018c559745a085861a4c1e90e6ce850da4b69c000425e40da37abc6906db01" exitCode=0
Jul 17 13:25:59.637889 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:25:59.637458    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771703-b4fbf" event={"ID":"88ec52a1-dbc0-4c2e-9db0-e83c0694fe9a","Type":"ContainerDied","Data":"52018c559745a085861a4c1e90e6ce850da4b69c000425e40da37abc6906db01"}
Jul 17 13:26:00.792590 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:00.792558    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771703-b4fbf"
Jul 17 13:26:00.940803 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:00.940708    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88ec52a1-dbc0-4c2e-9db0-e83c0694fe9a-maas-api-service-ca\") pod \"88ec52a1-dbc0-4c2e-9db0-e83c0694fe9a\" (UID: \"88ec52a1-dbc0-4c2e-9db0-e83c0694fe9a\") "
Jul 17 13:26:00.941163 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:00.941141    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88ec52a1-dbc0-4c2e-9db0-e83c0694fe9a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "88ec52a1-dbc0-4c2e-9db0-e83c0694fe9a" (UID: "88ec52a1-dbc0-4c2e-9db0-e83c0694fe9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:26:01.041631 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:01.041595    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88ec52a1-dbc0-4c2e-9db0-e83c0694fe9a-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:26:01.651127 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:01.651096    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771703-b4fbf"
Jul 17 13:26:01.651310 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:01.651131    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-771703-b4fbf" event={"ID":"88ec52a1-dbc0-4c2e-9db0-e83c0694fe9a","Type":"ContainerDied","Data":"adb52f20d766e4ae59e0076453d1450d2242b74af1ed6c72b9800f5092a7dfbc"}
Jul 17 13:26:01.651310 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:01.651176    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adb52f20d766e4ae59e0076453d1450d2242b74af1ed6c72b9800f5092a7dfbc"
Jul 17 13:26:06.840474 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:06.840438    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:06.938852 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:06.938810    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:08.159890 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:08.159847    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:08.259137 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:08.259104    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:08.441294 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:08.441213    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:09.064554 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:09.064520    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:09.164523 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:09.164478    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:10.876877 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:10.876828    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:12.685705 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:12.685667    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:12.856965 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:12.856928    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc"]
Jul 17 13:26:12.857444 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:12.857428    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="297acd52-45fb-4836-8eb8-429d9e947b14" containerName="revoke-keys"
Jul 17 13:26:12.857491 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:12.857446    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="297acd52-45fb-4836-8eb8-429d9e947b14" containerName="revoke-keys"
Jul 17 13:26:12.857491 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:12.857469    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88ec52a1-dbc0-4c2e-9db0-e83c0694fe9a" containerName="revoke-keys"
Jul 17 13:26:12.857491 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:12.857475    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="88ec52a1-dbc0-4c2e-9db0-e83c0694fe9a" containerName="revoke-keys"
Jul 17 13:26:12.857593 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:12.857543    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="88ec52a1-dbc0-4c2e-9db0-e83c0694fe9a" containerName="revoke-keys"
Jul 17 13:26:12.857593 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:12.857553    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="297acd52-45fb-4836-8eb8-429d9e947b14" containerName="revoke-keys"
Jul 17 13:26:12.860957 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:12.860943    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc"
Jul 17 13:26:12.864555 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:12.864515    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:26:12.867303 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:12.867278    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc"]
Jul 17 13:26:12.952317 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:12.952217    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d614a78-1da1-4f32-8783-8a82ed01e9d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc\" (UID: \"1d614a78-1da1-4f32-8783-8a82ed01e9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc"
Jul 17 13:26:13.052880 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:13.052842    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d614a78-1da1-4f32-8783-8a82ed01e9d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc\" (UID: \"1d614a78-1da1-4f32-8783-8a82ed01e9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc"
Jul 17 13:26:13.053483 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:13.053464    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d614a78-1da1-4f32-8783-8a82ed01e9d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc\" (UID: \"1d614a78-1da1-4f32-8783-8a82ed01e9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc"
Jul 17 13:26:13.174209 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:13.174164    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc"
Jul 17 13:26:13.321804 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:13.321775    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc"]
Jul 17 13:26:13.323625 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:26:13.323592    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d614a78_1da1_4f32_8783_8a82ed01e9d8.slice/crio-c46aad2eb15ff5645eabdb9c995738449dadf6295c7d1e62b452e61529485567 WatchSource:0}: Error finding container c46aad2eb15ff5645eabdb9c995738449dadf6295c7d1e62b452e61529485567: Status 404 returned error can't find the container with id c46aad2eb15ff5645eabdb9c995738449dadf6295c7d1e62b452e61529485567
Jul 17 13:26:13.728964 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:13.728924    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc" event={"ID":"1d614a78-1da1-4f32-8783-8a82ed01e9d8","Type":"ContainerStarted","Data":"2215facee02dbbc83aeb8dca33e07ea5c8eeec51b5df8673a121113f1c5ba69d"}
Jul 17 13:26:13.729492 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:13.729004    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc" event={"ID":"1d614a78-1da1-4f32-8783-8a82ed01e9d8","Type":"ContainerStarted","Data":"c46aad2eb15ff5645eabdb9c995738449dadf6295c7d1e62b452e61529485567"}
Jul 17 13:26:13.745571 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:13.745513    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc" podStartSLOduration=1.7454955559999998 podStartE2EDuration="1.745495556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:26:13.744414322 +0000 UTC m=+3817.875798983" watchObservedRunningTime="2026-07-17 13:26:13.745495556 +0000 UTC m=+3817.876880216"
Jul 17 13:26:14.737105 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:14.737069    2565 generic.go:356] "Generic (PLEG): container finished" podID="1d614a78-1da1-4f32-8783-8a82ed01e9d8" containerID="2215facee02dbbc83aeb8dca33e07ea5c8eeec51b5df8673a121113f1c5ba69d" exitCode=6
Jul 17 13:26:14.737518 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:14.737152    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc" event={"ID":"1d614a78-1da1-4f32-8783-8a82ed01e9d8","Type":"ContainerDied","Data":"2215facee02dbbc83aeb8dca33e07ea5c8eeec51b5df8673a121113f1c5ba69d"}
Jul 17 13:26:14.737518 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:14.737411    2565 scope.go:117] "RemoveContainer" containerID="2215facee02dbbc83aeb8dca33e07ea5c8eeec51b5df8673a121113f1c5ba69d"
Jul 17 13:26:15.746621 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:15.746531    2565 generic.go:356] "Generic (PLEG): container finished" podID="1d614a78-1da1-4f32-8783-8a82ed01e9d8" containerID="e96f6a9e4e27ea3ff760173faed7978c93e505962a7d32f71f2599069ef0e861" exitCode=6
Jul 17 13:26:15.747051 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:15.746616    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc" event={"ID":"1d614a78-1da1-4f32-8783-8a82ed01e9d8","Type":"ContainerDied","Data":"e96f6a9e4e27ea3ff760173faed7978c93e505962a7d32f71f2599069ef0e861"}
Jul 17 13:26:15.747051 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:15.746681    2565 scope.go:117] "RemoveContainer" containerID="2215facee02dbbc83aeb8dca33e07ea5c8eeec51b5df8673a121113f1c5ba69d"
Jul 17 13:26:15.747051 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:15.746896    2565 scope.go:117] "RemoveContainer" containerID="e96f6a9e4e27ea3ff760173faed7978c93e505962a7d32f71f2599069ef0e861"
Jul 17 13:26:15.747206 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:26:15.747136    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc_odh-ai-gateway-infra(1d614a78-1da1-4f32-8783-8a82ed01e9d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc" podUID="1d614a78-1da1-4f32-8783-8a82ed01e9d8"
Jul 17 13:26:16.755411 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:16.755385    2565 scope.go:117] "RemoveContainer" containerID="e96f6a9e4e27ea3ff760173faed7978c93e505962a7d32f71f2599069ef0e861"
Jul 17 13:26:16.755780 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:26:16.755571    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc_odh-ai-gateway-infra(1d614a78-1da1-4f32-8783-8a82ed01e9d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc" podUID="1d614a78-1da1-4f32-8783-8a82ed01e9d8"
Jul 17 13:26:18.376011 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:18.375947    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:18.476635 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:18.476594    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:23.426903 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:23.426863    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:23.681423 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:23.681316    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:24.589622 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:24.589589    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:24.688773 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:24.688738    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:31.396698 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:31.396661    2565 scope.go:117] "RemoveContainer" containerID="e96f6a9e4e27ea3ff760173faed7978c93e505962a7d32f71f2599069ef0e861"
Jul 17 13:26:31.848499 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:31.848458    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc" event={"ID":"1d614a78-1da1-4f32-8783-8a82ed01e9d8","Type":"ContainerStarted","Data":"c55d91a8c423211affc36fb89c9711a8f7fa00796a565d27ff4092a9328e4333"}
Jul 17 13:26:32.856524 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:32.856489    2565 generic.go:356] "Generic (PLEG): container finished" podID="1d614a78-1da1-4f32-8783-8a82ed01e9d8" containerID="c55d91a8c423211affc36fb89c9711a8f7fa00796a565d27ff4092a9328e4333" exitCode=6
Jul 17 13:26:32.857032 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:32.856563    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc" event={"ID":"1d614a78-1da1-4f32-8783-8a82ed01e9d8","Type":"ContainerDied","Data":"c55d91a8c423211affc36fb89c9711a8f7fa00796a565d27ff4092a9328e4333"}
Jul 17 13:26:32.857032 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:32.856624    2565 scope.go:117] "RemoveContainer" containerID="e96f6a9e4e27ea3ff760173faed7978c93e505962a7d32f71f2599069ef0e861"
Jul 17 13:26:32.857032 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:32.856856    2565 scope.go:117] "RemoveContainer" containerID="c55d91a8c423211affc36fb89c9711a8f7fa00796a565d27ff4092a9328e4333"
Jul 17 13:26:32.857179 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:26:32.857161    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc_odh-ai-gateway-infra(1d614a78-1da1-4f32-8783-8a82ed01e9d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc" podUID="1d614a78-1da1-4f32-8783-8a82ed01e9d8"
Jul 17 13:26:32.876070 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:32.876039    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc"]
Jul 17 13:26:34.011928 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:34.011899    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc"
Jul 17 13:26:34.045468 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:34.045440    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d614a78-1da1-4f32-8783-8a82ed01e9d8-maas-api-service-ca\") pod \"1d614a78-1da1-4f32-8783-8a82ed01e9d8\" (UID: \"1d614a78-1da1-4f32-8783-8a82ed01e9d8\") "
Jul 17 13:26:34.045916 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:34.045889    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d614a78-1da1-4f32-8783-8a82ed01e9d8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1d614a78-1da1-4f32-8783-8a82ed01e9d8" (UID: "1d614a78-1da1-4f32-8783-8a82ed01e9d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:26:34.147086 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:34.146971    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d614a78-1da1-4f32-8783-8a82ed01e9d8-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:26:34.391405 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:34.391373    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:34.872951 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:34.872923    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc"
Jul 17 13:26:34.873163 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:34.872962    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc" event={"ID":"1d614a78-1da1-4f32-8783-8a82ed01e9d8","Type":"ContainerDied","Data":"c46aad2eb15ff5645eabdb9c995738449dadf6295c7d1e62b452e61529485567"}
Jul 17 13:26:34.873163 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:34.873056    2565 scope.go:117] "RemoveContainer" containerID="c55d91a8c423211affc36fb89c9711a8f7fa00796a565d27ff4092a9328e4333"
Jul 17 13:26:34.891776 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:34.891746    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc"]
Jul 17 13:26:34.897952 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:34.897920    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-hjlcc"]
Jul 17 13:26:36.403669 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:36.403635    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d614a78-1da1-4f32-8783-8a82ed01e9d8" path="/var/lib/kubelet/pods/1d614a78-1da1-4f32-8783-8a82ed01e9d8/volumes"
Jul 17 13:26:38.923455 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:38.923414    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:44.962342 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:44.962301    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:46.487262 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:46.487226    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:49.670580 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:49.670547    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:49.913011 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:49.912953    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:50.297496 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:50.297455    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:50.629835 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:50.629750    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:50.728820 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:50.728785    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:53.522662 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:53.522630    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:53.626042 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:53.625978    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:53.733354 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:53.733298    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:53.843635 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:53.843597    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:54.334327 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:54.334273    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:54.658834 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:54.658749    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:58.033395 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:58.033308    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:58.377260 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:58.377229    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:58.712327 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:58.712235    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:26:58.810753 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:26:58.810714    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:27:01.545500 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:01.545466    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:27:01.654034 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:01.653954    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:27:01.759060 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:01.759022    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:27:02.395195 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:02.395162    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:27:02.495208 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:02.495172    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:27:05.506691 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:05.506650    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:27:06.017111 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:06.017075    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:27:10.294079 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:10.294039    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh"]
Jul 17 13:27:10.294543 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:10.294530    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d614a78-1da1-4f32-8783-8a82ed01e9d8" containerName="revoke-keys"
Jul 17 13:27:10.294543 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:10.294544    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d614a78-1da1-4f32-8783-8a82ed01e9d8" containerName="revoke-keys"
Jul 17 13:27:10.294623 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:10.294565    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d614a78-1da1-4f32-8783-8a82ed01e9d8" containerName="revoke-keys"
Jul 17 13:27:10.294623 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:10.294571    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d614a78-1da1-4f32-8783-8a82ed01e9d8" containerName="revoke-keys"
Jul 17 13:27:10.294685 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:10.294645    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d614a78-1da1-4f32-8783-8a82ed01e9d8" containerName="revoke-keys"
Jul 17 13:27:10.294685 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:10.294660    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d614a78-1da1-4f32-8783-8a82ed01e9d8" containerName="revoke-keys"
Jul 17 13:27:10.297810 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:10.297792    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh"
Jul 17 13:27:10.300951 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:10.300919    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:27:10.304486 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:10.304453    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh"]
Jul 17 13:27:10.398922 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:10.398893    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae9d6ab-1c76-420e-83ac-ba1c41e9eae3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh\" (UID: \"fae9d6ab-1c76-420e-83ac-ba1c41e9eae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh"
Jul 17 13:27:10.500239 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:10.500191    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae9d6ab-1c76-420e-83ac-ba1c41e9eae3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh\" (UID: \"fae9d6ab-1c76-420e-83ac-ba1c41e9eae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh"
Jul 17 13:27:10.500843 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:10.500822    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae9d6ab-1c76-420e-83ac-ba1c41e9eae3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh\" (UID: \"fae9d6ab-1c76-420e-83ac-ba1c41e9eae3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh"
Jul 17 13:27:10.611329 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:10.611289    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh"
Jul 17 13:27:10.760340 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:10.760314    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh"]
Jul 17 13:27:10.762559 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:27:10.762529    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae9d6ab_1c76_420e_83ac_ba1c41e9eae3.slice/crio-fb9b023b2c6fb4c5b03dfab29f16b91f3cb804143989049ef43f0f4b27596656 WatchSource:0}: Error finding container fb9b023b2c6fb4c5b03dfab29f16b91f3cb804143989049ef43f0f4b27596656: Status 404 returned error can't find the container with id fb9b023b2c6fb4c5b03dfab29f16b91f3cb804143989049ef43f0f4b27596656
Jul 17 13:27:11.097100 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:11.097063    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh" event={"ID":"fae9d6ab-1c76-420e-83ac-ba1c41e9eae3","Type":"ContainerStarted","Data":"e1ba2dc1cc55736d34553ff0eebf26a44db1689de9b86ee3bed5137fe34225bf"}
Jul 17 13:27:11.097272 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:11.097113    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh" event={"ID":"fae9d6ab-1c76-420e-83ac-ba1c41e9eae3","Type":"ContainerStarted","Data":"fb9b023b2c6fb4c5b03dfab29f16b91f3cb804143989049ef43f0f4b27596656"}
Jul 17 13:27:11.116264 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:11.116204    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh" podStartSLOduration=1.116189639 podStartE2EDuration="1.116189639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:27:11.114035375 +0000 UTC m=+3875.245420035" watchObservedRunningTime="2026-07-17 13:27:11.116189639 +0000 UTC m=+3875.247574340"
Jul 17 13:27:12.105465 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:12.105373    2565 generic.go:356] "Generic (PLEG): container finished" podID="fae9d6ab-1c76-420e-83ac-ba1c41e9eae3" containerID="e1ba2dc1cc55736d34553ff0eebf26a44db1689de9b86ee3bed5137fe34225bf" exitCode=6
Jul 17 13:27:12.105465 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:12.105454    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh" event={"ID":"fae9d6ab-1c76-420e-83ac-ba1c41e9eae3","Type":"ContainerDied","Data":"e1ba2dc1cc55736d34553ff0eebf26a44db1689de9b86ee3bed5137fe34225bf"}
Jul 17 13:27:12.105899 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:12.105726    2565 scope.go:117] "RemoveContainer" containerID="e1ba2dc1cc55736d34553ff0eebf26a44db1689de9b86ee3bed5137fe34225bf"
Jul 17 13:27:12.906225 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:12.906187    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz"]
Jul 17 13:27:12.906703 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:12.906688    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d614a78-1da1-4f32-8783-8a82ed01e9d8" containerName="revoke-keys"
Jul 17 13:27:12.906759 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:12.906708    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d614a78-1da1-4f32-8783-8a82ed01e9d8" containerName="revoke-keys"
Jul 17 13:27:12.906808 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:12.906801    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d614a78-1da1-4f32-8783-8a82ed01e9d8" containerName="revoke-keys"
Jul 17 13:27:12.910179 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:12.910162    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz"
Jul 17 13:27:12.918641 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:12.918607    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz"]
Jul 17 13:27:13.024364 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:13.024315    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc308298-bcde-459d-afac-685bb29a1d0d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz\" (UID: \"bc308298-bcde-459d-afac-685bb29a1d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz"
Jul 17 13:27:13.116226 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:13.116132    2565 generic.go:356] "Generic (PLEG): container finished" podID="fae9d6ab-1c76-420e-83ac-ba1c41e9eae3" containerID="3cce4abc9eac8ef2a831a73f204d22e51c6fc936818aacf8a1920fe46a70ab6e" exitCode=6
Jul 17 13:27:13.116226 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:13.116186    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh" event={"ID":"fae9d6ab-1c76-420e-83ac-ba1c41e9eae3","Type":"ContainerDied","Data":"3cce4abc9eac8ef2a831a73f204d22e51c6fc936818aacf8a1920fe46a70ab6e"}
Jul 17 13:27:13.116746 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:13.116246    2565 scope.go:117] "RemoveContainer" containerID="e1ba2dc1cc55736d34553ff0eebf26a44db1689de9b86ee3bed5137fe34225bf"
Jul 17 13:27:13.116746 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:13.116427    2565 scope.go:117] "RemoveContainer" containerID="3cce4abc9eac8ef2a831a73f204d22e51c6fc936818aacf8a1920fe46a70ab6e"
Jul 17 13:27:13.116746 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:27:13.116702    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh_odh-ai-gateway-infra(fae9d6ab-1c76-420e-83ac-ba1c41e9eae3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh" podUID="fae9d6ab-1c76-420e-83ac-ba1c41e9eae3"
Jul 17 13:27:13.125687 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:13.125651    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc308298-bcde-459d-afac-685bb29a1d0d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz\" (UID: \"bc308298-bcde-459d-afac-685bb29a1d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz"
Jul 17 13:27:13.126331 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:13.126306    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc308298-bcde-459d-afac-685bb29a1d0d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz\" (UID: \"bc308298-bcde-459d-afac-685bb29a1d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz"
Jul 17 13:27:13.223881 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:13.223844    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz"
Jul 17 13:27:13.374113 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:13.374083    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz"]
Jul 17 13:27:13.375858 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:27:13.375829    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc308298_bcde_459d_afac_685bb29a1d0d.slice/crio-71a25f5d30a7ce98e04291a0adea2f112c9ed1aaf9ac41b238c881bc0abc0355 WatchSource:0}: Error finding container 71a25f5d30a7ce98e04291a0adea2f112c9ed1aaf9ac41b238c881bc0abc0355: Status 404 returned error can't find the container with id 71a25f5d30a7ce98e04291a0adea2f112c9ed1aaf9ac41b238c881bc0abc0355
Jul 17 13:27:14.128036 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:14.125284    2565 generic.go:356] "Generic (PLEG): container finished" podID="bc308298-bcde-459d-afac-685bb29a1d0d" containerID="beaf26265840c95f8047a9cca9ba1b26aa3c362d905b9498884f9ce8abdb53eb" exitCode=6
Jul 17 13:27:14.128036 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:14.125520    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz" event={"ID":"bc308298-bcde-459d-afac-685bb29a1d0d","Type":"ContainerDied","Data":"beaf26265840c95f8047a9cca9ba1b26aa3c362d905b9498884f9ce8abdb53eb"}
Jul 17 13:27:14.128036 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:14.125575    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz" event={"ID":"bc308298-bcde-459d-afac-685bb29a1d0d","Type":"ContainerStarted","Data":"71a25f5d30a7ce98e04291a0adea2f112c9ed1aaf9ac41b238c881bc0abc0355"}
Jul 17 13:27:14.128036 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:14.125906    2565 scope.go:117] "RemoveContainer" containerID="beaf26265840c95f8047a9cca9ba1b26aa3c362d905b9498884f9ce8abdb53eb"
Jul 17 13:27:14.129863 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:14.129842    2565 scope.go:117] "RemoveContainer" containerID="3cce4abc9eac8ef2a831a73f204d22e51c6fc936818aacf8a1920fe46a70ab6e"
Jul 17 13:27:14.130118 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:27:14.130094    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh_odh-ai-gateway-infra(fae9d6ab-1c76-420e-83ac-ba1c41e9eae3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh" podUID="fae9d6ab-1c76-420e-83ac-ba1c41e9eae3"
Jul 17 13:27:15.127927 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:15.127823    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:27:15.138334 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:15.138304    2565 generic.go:356] "Generic (PLEG): container finished" podID="bc308298-bcde-459d-afac-685bb29a1d0d" containerID="7a8d6be55f9b273bc94aa3b847410ea3c8b5f180f77b0a597b66b6b327b5e116" exitCode=6
Jul 17 13:27:15.138745 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:15.138389    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz" event={"ID":"bc308298-bcde-459d-afac-685bb29a1d0d","Type":"ContainerDied","Data":"7a8d6be55f9b273bc94aa3b847410ea3c8b5f180f77b0a597b66b6b327b5e116"}
Jul 17 13:27:15.138745 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:15.138482    2565 scope.go:117] "RemoveContainer" containerID="beaf26265840c95f8047a9cca9ba1b26aa3c362d905b9498884f9ce8abdb53eb"
Jul 17 13:27:15.138745 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:15.138592    2565 scope.go:117] "RemoveContainer" containerID="7a8d6be55f9b273bc94aa3b847410ea3c8b5f180f77b0a597b66b6b327b5e116"
Jul 17 13:27:15.138903 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:27:15.138825    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz_odh-ai-gateway-infra(bc308298-bcde-459d-afac-685bb29a1d0d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz" podUID="bc308298-bcde-459d-afac-685bb29a1d0d"
Jul 17 13:27:15.228479 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:15.228443    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:27:16.147351 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:16.147324    2565 scope.go:117] "RemoveContainer" containerID="7a8d6be55f9b273bc94aa3b847410ea3c8b5f180f77b0a597b66b6b327b5e116"
Jul 17 13:27:16.147751 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:27:16.147549    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz_odh-ai-gateway-infra(bc308298-bcde-459d-afac-685bb29a1d0d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz" podUID="bc308298-bcde-459d-afac-685bb29a1d0d"
Jul 17 13:27:21.665249 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:21.665209    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:27:23.356074 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:23.356042    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:27:23.456390 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:23.456354    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:27:26.400891 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:26.400836    2565 scope.go:117] "RemoveContainer" containerID="3cce4abc9eac8ef2a831a73f204d22e51c6fc936818aacf8a1920fe46a70ab6e"
Jul 17 13:27:27.217189 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:27.217150    2565 generic.go:356] "Generic (PLEG): container finished" podID="fae9d6ab-1c76-420e-83ac-ba1c41e9eae3" containerID="90fc1ed73cf4ec6ca83d87d43993aa55f258f2a6c179e90ba76ede33630024f4" exitCode=6
Jul 17 13:27:27.217391 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:27.217222    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh" event={"ID":"fae9d6ab-1c76-420e-83ac-ba1c41e9eae3","Type":"ContainerDied","Data":"90fc1ed73cf4ec6ca83d87d43993aa55f258f2a6c179e90ba76ede33630024f4"}
Jul 17 13:27:27.217391 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:27.217300    2565 scope.go:117] "RemoveContainer" containerID="3cce4abc9eac8ef2a831a73f204d22e51c6fc936818aacf8a1920fe46a70ab6e"
Jul 17 13:27:27.217687 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:27.217663    2565 scope.go:117] "RemoveContainer" containerID="90fc1ed73cf4ec6ca83d87d43993aa55f258f2a6c179e90ba76ede33630024f4"
Jul 17 13:27:27.217936 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:27:27.217915    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh_odh-ai-gateway-infra(fae9d6ab-1c76-420e-83ac-ba1c41e9eae3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh" podUID="fae9d6ab-1c76-420e-83ac-ba1c41e9eae3"
Jul 17 13:27:28.251098 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:28.251066    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh"]
Jul 17 13:27:28.396705 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:28.396678    2565 scope.go:117] "RemoveContainer" containerID="7a8d6be55f9b273bc94aa3b847410ea3c8b5f180f77b0a597b66b6b327b5e116"
Jul 17 13:27:28.404721 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:28.404696    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh"
Jul 17 13:27:28.475406 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:28.475362    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae9d6ab-1c76-420e-83ac-ba1c41e9eae3-maas-api-service-ca\") pod \"fae9d6ab-1c76-420e-83ac-ba1c41e9eae3\" (UID: \"fae9d6ab-1c76-420e-83ac-ba1c41e9eae3\") "
Jul 17 13:27:28.475745 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:28.475721    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae9d6ab-1c76-420e-83ac-ba1c41e9eae3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fae9d6ab-1c76-420e-83ac-ba1c41e9eae3" (UID: "fae9d6ab-1c76-420e-83ac-ba1c41e9eae3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:27:28.576848 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:28.576816    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fae9d6ab-1c76-420e-83ac-ba1c41e9eae3-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:27:29.233933 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:29.233907    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh"
Jul 17 13:27:29.234151 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:29.233947    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh" event={"ID":"fae9d6ab-1c76-420e-83ac-ba1c41e9eae3","Type":"ContainerDied","Data":"fb9b023b2c6fb4c5b03dfab29f16b91f3cb804143989049ef43f0f4b27596656"}
Jul 17 13:27:29.234151 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:29.234060    2565 scope.go:117] "RemoveContainer" containerID="90fc1ed73cf4ec6ca83d87d43993aa55f258f2a6c179e90ba76ede33630024f4"
Jul 17 13:27:29.236054 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:29.236024    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz" event={"ID":"bc308298-bcde-459d-afac-685bb29a1d0d","Type":"ContainerDied","Data":"f956bbffd8df22093d5a792c5ceeff457af3cf6ac0940b4b6d74b9dc09c2ea10"}
Jul 17 13:27:29.236177 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:29.236060    2565 generic.go:356] "Generic (PLEG): container finished" podID="bc308298-bcde-459d-afac-685bb29a1d0d" containerID="f956bbffd8df22093d5a792c5ceeff457af3cf6ac0940b4b6d74b9dc09c2ea10" exitCode=6
Jul 17 13:27:29.236343 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:29.236328    2565 scope.go:117] "RemoveContainer" containerID="f956bbffd8df22093d5a792c5ceeff457af3cf6ac0940b4b6d74b9dc09c2ea10"
Jul 17 13:27:29.236536 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:27:29.236519    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz_odh-ai-gateway-infra(bc308298-bcde-459d-afac-685bb29a1d0d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz" podUID="bc308298-bcde-459d-afac-685bb29a1d0d"
Jul 17 13:27:29.275512 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:29.275348    2565 scope.go:117] "RemoveContainer" containerID="7a8d6be55f9b273bc94aa3b847410ea3c8b5f180f77b0a597b66b6b327b5e116"
Jul 17 13:27:29.285632 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:29.285603    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh"]
Jul 17 13:27:29.289372 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:29.289348    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-pn7xh"]
Jul 17 13:27:30.262184 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:30.262146    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz"]
Jul 17 13:27:30.404521 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:30.404491    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fae9d6ab-1c76-420e-83ac-ba1c41e9eae3" path="/var/lib/kubelet/pods/fae9d6ab-1c76-420e-83ac-ba1c41e9eae3/volumes"
Jul 17 13:27:30.416116 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:30.416079    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz"
Jul 17 13:27:30.494689 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:30.494648    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc308298-bcde-459d-afac-685bb29a1d0d-maas-api-service-ca\") pod \"bc308298-bcde-459d-afac-685bb29a1d0d\" (UID: \"bc308298-bcde-459d-afac-685bb29a1d0d\") "
Jul 17 13:27:30.495086 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:30.495061    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc308298-bcde-459d-afac-685bb29a1d0d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bc308298-bcde-459d-afac-685bb29a1d0d" (UID: "bc308298-bcde-459d-afac-685bb29a1d0d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:27:30.595861 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:30.595823    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc308298-bcde-459d-afac-685bb29a1d0d-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:27:31.253099 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:31.253059    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz"
Jul 17 13:27:31.253099 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:31.253091    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz" event={"ID":"bc308298-bcde-459d-afac-685bb29a1d0d","Type":"ContainerDied","Data":"71a25f5d30a7ce98e04291a0adea2f112c9ed1aaf9ac41b238c881bc0abc0355"}
Jul 17 13:27:31.253349 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:31.253154    2565 scope.go:117] "RemoveContainer" containerID="f956bbffd8df22093d5a792c5ceeff457af3cf6ac0940b4b6d74b9dc09c2ea10"
Jul 17 13:27:31.275721 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:31.275687    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz"]
Jul 17 13:27:31.281243 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:31.281216    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-98wxz"]
Jul 17 13:27:32.403535 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:32.403502    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc308298-bcde-459d-afac-685bb29a1d0d" path="/var/lib/kubelet/pods/bc308298-bcde-459d-afac-685bb29a1d0d/volumes"
Jul 17 13:27:36.779066 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:36.779040    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 13:27:36.784410 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:36.784381    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 13:27:36.796603 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:36.796580    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 13:27:36.801824 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:36.801805    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 13:27:45.027001 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.026947    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:27:45.077176 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.077134    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cngdx"]
Jul 17 13:27:45.077701 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.077682    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc308298-bcde-459d-afac-685bb29a1d0d" containerName="revoke-keys"
Jul 17 13:27:45.077701 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.077700    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc308298-bcde-459d-afac-685bb29a1d0d" containerName="revoke-keys"
Jul 17 13:27:45.077849 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.077711    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc308298-bcde-459d-afac-685bb29a1d0d" containerName="revoke-keys"
Jul 17 13:27:45.077849 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.077716    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc308298-bcde-459d-afac-685bb29a1d0d" containerName="revoke-keys"
Jul 17 13:27:45.077849 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.077724    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae9d6ab-1c76-420e-83ac-ba1c41e9eae3" containerName="revoke-keys"
Jul 17 13:27:45.077849 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.077730    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae9d6ab-1c76-420e-83ac-ba1c41e9eae3" containerName="revoke-keys"
Jul 17 13:27:45.077849 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.077736    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae9d6ab-1c76-420e-83ac-ba1c41e9eae3" containerName="revoke-keys"
Jul 17 13:27:45.077849 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.077741    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae9d6ab-1c76-420e-83ac-ba1c41e9eae3" containerName="revoke-keys"
Jul 17 13:27:45.077849 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.077750    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc308298-bcde-459d-afac-685bb29a1d0d" containerName="revoke-keys"
Jul 17 13:27:45.077849 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.077755    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc308298-bcde-459d-afac-685bb29a1d0d" containerName="revoke-keys"
Jul 17 13:27:45.077849 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.077762    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae9d6ab-1c76-420e-83ac-ba1c41e9eae3" containerName="revoke-keys"
Jul 17 13:27:45.077849 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.077768    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae9d6ab-1c76-420e-83ac-ba1c41e9eae3" containerName="revoke-keys"
Jul 17 13:27:45.077849 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.077836    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc308298-bcde-459d-afac-685bb29a1d0d" containerName="revoke-keys"
Jul 17 13:27:45.077849 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.077843    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc308298-bcde-459d-afac-685bb29a1d0d" containerName="revoke-keys"
Jul 17 13:27:45.077849 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.077850    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fae9d6ab-1c76-420e-83ac-ba1c41e9eae3" containerName="revoke-keys"
Jul 17 13:27:45.077849 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.077855    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fae9d6ab-1c76-420e-83ac-ba1c41e9eae3" containerName="revoke-keys"
Jul 17 13:27:45.085027 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.084308    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cngdx"
Jul 17 13:27:45.094317 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.094283    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cngdx"]
Jul 17 13:27:45.240160 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.240125    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6qzm\" (UniqueName: \"kubernetes.io/projected/4fbab352-87f6-47aa-8acf-ff01f07a685a-kube-api-access-j6qzm\") pod \"kuadrant-operator-catalog-cngdx\" (UID: \"4fbab352-87f6-47aa-8acf-ff01f07a685a\") " pod="kuadrant-system/kuadrant-operator-catalog-cngdx"
Jul 17 13:27:45.341480 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.341447    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6qzm\" (UniqueName: \"kubernetes.io/projected/4fbab352-87f6-47aa-8acf-ff01f07a685a-kube-api-access-j6qzm\") pod \"kuadrant-operator-catalog-cngdx\" (UID: \"4fbab352-87f6-47aa-8acf-ff01f07a685a\") " pod="kuadrant-system/kuadrant-operator-catalog-cngdx"
Jul 17 13:27:45.351109 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.351081    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6qzm\" (UniqueName: \"kubernetes.io/projected/4fbab352-87f6-47aa-8acf-ff01f07a685a-kube-api-access-j6qzm\") pod \"kuadrant-operator-catalog-cngdx\" (UID: \"4fbab352-87f6-47aa-8acf-ff01f07a685a\") " pod="kuadrant-system/kuadrant-operator-catalog-cngdx"
Jul 17 13:27:45.399645 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.399609    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cngdx"
Jul 17 13:27:45.547802 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.547772    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cngdx"]
Jul 17 13:27:45.549495 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:27:45.549469    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fbab352_87f6_47aa_8acf_ff01f07a685a.slice/crio-90eaf9be0761f443eecaf8b449ddc8620def11c009c9a94a60b468b3498cedb8 WatchSource:0}: Error finding container 90eaf9be0761f443eecaf8b449ddc8620def11c009c9a94a60b468b3498cedb8: Status 404 returned error can't find the container with id 90eaf9be0761f443eecaf8b449ddc8620def11c009c9a94a60b468b3498cedb8
Jul 17 13:27:45.550867 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:45.550847    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 13:27:46.353230 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:46.353194    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cngdx" event={"ID":"4fbab352-87f6-47aa-8acf-ff01f07a685a","Type":"ContainerStarted","Data":"90eaf9be0761f443eecaf8b449ddc8620def11c009c9a94a60b468b3498cedb8"}
Jul 17 13:27:47.362455 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:47.362416    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cngdx" event={"ID":"4fbab352-87f6-47aa-8acf-ff01f07a685a","Type":"ContainerStarted","Data":"3a5179de6e8f4c61f3863d8a100bcc39589968fbf76328c17d8eabf58749b60c"}
Jul 17 13:27:47.381002 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:47.380930    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-cngdx" podStartSLOduration=1.541211785 podStartE2EDuration="2.380913677s" totalImagesPullingTime="839.701892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:27:47.378242717 +0000 UTC m=+3911.509627401" watchObservedRunningTime="2026-07-17 13:27:47.380913677 +0000 UTC m=+3911.512298345"
Jul 17 13:27:55.400406 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:55.400360    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-cngdx"
Jul 17 13:27:55.400834 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:55.400434    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-cngdx"
Jul 17 13:27:55.426217 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:55.426192    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-cngdx"
Jul 17 13:27:55.450304 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:55.450272    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-cngdx"
Jul 17 13:27:55.655372 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:55.655282    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cngdx"]
Jul 17 13:27:57.428220 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:57.428155    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-cngdx" podUID="4fbab352-87f6-47aa-8acf-ff01f07a685a" containerName="registry-server" containerID="cri-o://3a5179de6e8f4c61f3863d8a100bcc39589968fbf76328c17d8eabf58749b60c" gracePeriod=2
Jul 17 13:27:57.684801 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:57.684735    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cngdx"
Jul 17 13:27:57.747923 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:57.747889    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6qzm\" (UniqueName: \"kubernetes.io/projected/4fbab352-87f6-47aa-8acf-ff01f07a685a-kube-api-access-j6qzm\") pod \"4fbab352-87f6-47aa-8acf-ff01f07a685a\" (UID: \"4fbab352-87f6-47aa-8acf-ff01f07a685a\") "
Jul 17 13:27:57.750161 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:57.750131    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbab352-87f6-47aa-8acf-ff01f07a685a-kube-api-access-j6qzm" (OuterVolumeSpecName: "kube-api-access-j6qzm") pod "4fbab352-87f6-47aa-8acf-ff01f07a685a" (UID: "4fbab352-87f6-47aa-8acf-ff01f07a685a"). InnerVolumeSpecName "kube-api-access-j6qzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:27:57.849430 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:57.849382    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6qzm\" (UniqueName: \"kubernetes.io/projected/4fbab352-87f6-47aa-8acf-ff01f07a685a-kube-api-access-j6qzm\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:27:58.435790 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:58.435753    2565 generic.go:356] "Generic (PLEG): container finished" podID="4fbab352-87f6-47aa-8acf-ff01f07a685a" containerID="3a5179de6e8f4c61f3863d8a100bcc39589968fbf76328c17d8eabf58749b60c" exitCode=0
Jul 17 13:27:58.436223 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:58.435799    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cngdx"
Jul 17 13:27:58.436223 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:58.435835    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cngdx" event={"ID":"4fbab352-87f6-47aa-8acf-ff01f07a685a","Type":"ContainerDied","Data":"3a5179de6e8f4c61f3863d8a100bcc39589968fbf76328c17d8eabf58749b60c"}
Jul 17 13:27:58.436223 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:58.435884    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cngdx" event={"ID":"4fbab352-87f6-47aa-8acf-ff01f07a685a","Type":"ContainerDied","Data":"90eaf9be0761f443eecaf8b449ddc8620def11c009c9a94a60b468b3498cedb8"}
Jul 17 13:27:58.436223 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:58.435915    2565 scope.go:117] "RemoveContainer" containerID="3a5179de6e8f4c61f3863d8a100bcc39589968fbf76328c17d8eabf58749b60c"
Jul 17 13:27:58.446396 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:58.446380    2565 scope.go:117] "RemoveContainer" containerID="3a5179de6e8f4c61f3863d8a100bcc39589968fbf76328c17d8eabf58749b60c"
Jul 17 13:27:58.446670 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:27:58.446651    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a5179de6e8f4c61f3863d8a100bcc39589968fbf76328c17d8eabf58749b60c\": container with ID starting with 3a5179de6e8f4c61f3863d8a100bcc39589968fbf76328c17d8eabf58749b60c not found: ID does not exist" containerID="3a5179de6e8f4c61f3863d8a100bcc39589968fbf76328c17d8eabf58749b60c"
Jul 17 13:27:58.446717 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:58.446680    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a5179de6e8f4c61f3863d8a100bcc39589968fbf76328c17d8eabf58749b60c"} err="failed to get container status \"3a5179de6e8f4c61f3863d8a100bcc39589968fbf76328c17d8eabf58749b60c\": rpc error: code = NotFound desc = could not find container \"3a5179de6e8f4c61f3863d8a100bcc39589968fbf76328c17d8eabf58749b60c\": container with ID starting with 3a5179de6e8f4c61f3863d8a100bcc39589968fbf76328c17d8eabf58749b60c not found: ID does not exist"
Jul 17 13:27:58.453350 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:58.453313    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cngdx"]
Jul 17 13:27:58.456052 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:58.456034    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cngdx"]
Jul 17 13:27:59.062210 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:59.062169    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx"]
Jul 17 13:27:59.062689 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:59.062671    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fbab352-87f6-47aa-8acf-ff01f07a685a" containerName="registry-server"
Jul 17 13:27:59.062779 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:59.062693    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fbab352-87f6-47aa-8acf-ff01f07a685a" containerName="registry-server"
Jul 17 13:27:59.062855 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:59.062842    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fbab352-87f6-47aa-8acf-ff01f07a685a" containerName="registry-server"
Jul 17 13:27:59.062914 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:59.062863    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc308298-bcde-459d-afac-685bb29a1d0d" containerName="revoke-keys"
Jul 17 13:27:59.062914 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:59.062874    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fae9d6ab-1c76-420e-83ac-ba1c41e9eae3" containerName="revoke-keys"
Jul 17 13:27:59.067293 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:59.067270    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx"
Jul 17 13:27:59.070226 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:59.070193    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:27:59.075164 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:59.075140    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx"]
Jul 17 13:27:59.162896 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:59.162857    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx\" (UID: \"64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx"
Jul 17 13:27:59.264433 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:59.264389    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx\" (UID: \"64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx"
Jul 17 13:27:59.265075 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:59.265055    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx\" (UID: \"64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx"
Jul 17 13:27:59.379749 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:59.379662    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx"
Jul 17 13:27:59.531097 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:27:59.531070    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx"]
Jul 17 13:27:59.532562 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:27:59.532537    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64aa95a7_ea2f_4c38_a9c0_4a89c0ae6b25.slice/crio-bbad814d552b17900c1172d6abd2f4af1fdc84d3e97466847e7213150e957de9 WatchSource:0}: Error finding container bbad814d552b17900c1172d6abd2f4af1fdc84d3e97466847e7213150e957de9: Status 404 returned error can't find the container with id bbad814d552b17900c1172d6abd2f4af1fdc84d3e97466847e7213150e957de9
Jul 17 13:28:00.338011 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:00.337965    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb"]
Jul 17 13:28:00.341610 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:00.341592    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb"
Jul 17 13:28:00.350213 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:00.350185    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb"]
Jul 17 13:28:00.373885 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:00.373853    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19338a8c-78db-48e3-8668-796209c087c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb\" (UID: \"19338a8c-78db-48e3-8668-796209c087c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb"
Jul 17 13:28:00.402941 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:00.402909    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fbab352-87f6-47aa-8acf-ff01f07a685a" path="/var/lib/kubelet/pods/4fbab352-87f6-47aa-8acf-ff01f07a685a/volumes"
Jul 17 13:28:00.453609 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:00.453573    2565 generic.go:356] "Generic (PLEG): container finished" podID="64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25" containerID="1fbec99e01fd048b9b8a58416820cec5059fe456d560e1dad1aaa0b25e69d5ed" exitCode=6
Jul 17 13:28:00.453800 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:00.453662    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx" event={"ID":"64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25","Type":"ContainerDied","Data":"1fbec99e01fd048b9b8a58416820cec5059fe456d560e1dad1aaa0b25e69d5ed"}
Jul 17 13:28:00.453800 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:00.453726    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx" event={"ID":"64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25","Type":"ContainerStarted","Data":"bbad814d552b17900c1172d6abd2f4af1fdc84d3e97466847e7213150e957de9"}
Jul 17 13:28:00.453884 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:00.453852    2565 scope.go:117] "RemoveContainer" containerID="1fbec99e01fd048b9b8a58416820cec5059fe456d560e1dad1aaa0b25e69d5ed"
Jul 17 13:28:00.475144 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:00.475116    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19338a8c-78db-48e3-8668-796209c087c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb\" (UID: \"19338a8c-78db-48e3-8668-796209c087c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb"
Jul 17 13:28:00.475695 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:00.475674    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19338a8c-78db-48e3-8668-796209c087c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb\" (UID: \"19338a8c-78db-48e3-8668-796209c087c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb"
Jul 17 13:28:00.655532 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:00.655496    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb"
Jul 17 13:28:00.809115 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:00.809089    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb"]
Jul 17 13:28:00.811040 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:28:00.811008    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19338a8c_78db_48e3_8668_796209c087c1.slice/crio-9e86adb9520b1648ccecdc66851c333c06fbbb6debcc843bd5443be5829b6f00 WatchSource:0}: Error finding container 9e86adb9520b1648ccecdc66851c333c06fbbb6debcc843bd5443be5829b6f00: Status 404 returned error can't find the container with id 9e86adb9520b1648ccecdc66851c333c06fbbb6debcc843bd5443be5829b6f00
Jul 17 13:28:01.462215 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:01.462177    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb" event={"ID":"19338a8c-78db-48e3-8668-796209c087c1","Type":"ContainerStarted","Data":"67301fb4cc224b00d7b69d5d4043bf07438501cbd11d0033ee978ff05eb1a1b3"}
Jul 17 13:28:01.462393 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:01.462247    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb" event={"ID":"19338a8c-78db-48e3-8668-796209c087c1","Type":"ContainerStarted","Data":"9e86adb9520b1648ccecdc66851c333c06fbbb6debcc843bd5443be5829b6f00"}
Jul 17 13:28:01.464296 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:01.464271    2565 generic.go:356] "Generic (PLEG): container finished" podID="64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25" containerID="e7e1e01f14c145fb35614165f36897ec5615be09320a866192c8145a55b2c32a" exitCode=6
Jul 17 13:28:01.464413 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:01.464351    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx" event={"ID":"64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25","Type":"ContainerDied","Data":"e7e1e01f14c145fb35614165f36897ec5615be09320a866192c8145a55b2c32a"}
Jul 17 13:28:01.464413 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:01.464402    2565 scope.go:117] "RemoveContainer" containerID="1fbec99e01fd048b9b8a58416820cec5059fe456d560e1dad1aaa0b25e69d5ed"
Jul 17 13:28:01.464512 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:01.464471    2565 scope.go:117] "RemoveContainer" containerID="e7e1e01f14c145fb35614165f36897ec5615be09320a866192c8145a55b2c32a"
Jul 17 13:28:01.464719 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:28:01.464695    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx_odh-ai-gateway-infra(64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx" podUID="64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25"
Jul 17 13:28:01.480672 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:01.480630    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb" podStartSLOduration=1.48061622 podStartE2EDuration="1.48061622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:28:01.478429221 +0000 UTC m=+3925.609813882" watchObservedRunningTime="2026-07-17 13:28:01.48061622 +0000 UTC m=+3925.612000880"
Jul 17 13:28:01.578364 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:01.578333    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8"]
Jul 17 13:28:01.581994 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:01.581970    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8"
Jul 17 13:28:01.591402 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:01.591371    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8"]
Jul 17 13:28:01.687085 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:01.687046    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72294120-389e-4270-bd7c-1ca994800275-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8\" (UID: \"72294120-389e-4270-bd7c-1ca994800275\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8"
Jul 17 13:28:01.787840 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:01.787740    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72294120-389e-4270-bd7c-1ca994800275-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8\" (UID: \"72294120-389e-4270-bd7c-1ca994800275\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8"
Jul 17 13:28:01.788448 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:01.788425    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72294120-389e-4270-bd7c-1ca994800275-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8\" (UID: \"72294120-389e-4270-bd7c-1ca994800275\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8"
Jul 17 13:28:01.903727 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:01.903691    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8"
Jul 17 13:28:02.048732 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:02.048700    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8"]
Jul 17 13:28:02.049733 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:28:02.049709    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72294120_389e_4270_bd7c_1ca994800275.slice/crio-7e293057e1548c90a70e1d91b271e0eeb413f7c12d3a5465b852a2e6cb32990a WatchSource:0}: Error finding container 7e293057e1548c90a70e1d91b271e0eeb413f7c12d3a5465b852a2e6cb32990a: Status 404 returned error can't find the container with id 7e293057e1548c90a70e1d91b271e0eeb413f7c12d3a5465b852a2e6cb32990a
Jul 17 13:28:02.472203 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:02.472163    2565 generic.go:356] "Generic (PLEG): container finished" podID="19338a8c-78db-48e3-8668-796209c087c1" containerID="67301fb4cc224b00d7b69d5d4043bf07438501cbd11d0033ee978ff05eb1a1b3" exitCode=6
Jul 17 13:28:02.472347 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:02.472246    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb" event={"ID":"19338a8c-78db-48e3-8668-796209c087c1","Type":"ContainerDied","Data":"67301fb4cc224b00d7b69d5d4043bf07438501cbd11d0033ee978ff05eb1a1b3"}
Jul 17 13:28:02.472493 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:02.472472    2565 scope.go:117] "RemoveContainer" containerID="67301fb4cc224b00d7b69d5d4043bf07438501cbd11d0033ee978ff05eb1a1b3"
Jul 17 13:28:02.474206 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:02.474173    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8" event={"ID":"72294120-389e-4270-bd7c-1ca994800275","Type":"ContainerStarted","Data":"8249cd40cbf187c557e7c95181e89a1558991e5df86c7a355d4a4e0fcce13832"}
Jul 17 13:28:02.474341 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:02.474226    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8" event={"ID":"72294120-389e-4270-bd7c-1ca994800275","Type":"ContainerStarted","Data":"7e293057e1548c90a70e1d91b271e0eeb413f7c12d3a5465b852a2e6cb32990a"}
Jul 17 13:28:02.476375 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:02.476357    2565 scope.go:117] "RemoveContainer" containerID="e7e1e01f14c145fb35614165f36897ec5615be09320a866192c8145a55b2c32a"
Jul 17 13:28:02.476605 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:28:02.476583    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx_odh-ai-gateway-infra(64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx" podUID="64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25"
Jul 17 13:28:02.525757 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:02.525707    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8" podStartSLOduration=1.5256915690000001 podStartE2EDuration="1.525691569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:28:02.523047093 +0000 UTC m=+3926.654431774" watchObservedRunningTime="2026-07-17 13:28:02.525691569 +0000 UTC m=+3926.657076266"
Jul 17 13:28:02.944479 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:02.944383    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx"]
Jul 17 13:28:02.948173 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:02.948154    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx"
Jul 17 13:28:02.954977 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:02.954947    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx"]
Jul 17 13:28:02.999555 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:02.999513    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx\" (UID: \"4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx"
Jul 17 13:28:03.100924 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:03.100871    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx\" (UID: \"4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx"
Jul 17 13:28:03.101619 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:03.101592    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx\" (UID: \"4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx"
Jul 17 13:28:03.261069 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:03.261026    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx"
Jul 17 13:28:03.415691 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:03.415660    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx"]
Jul 17 13:28:03.416918 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:28:03.416889    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6497d8_4fd6_479b_ba90_ad6d3bb19c6a.slice/crio-b0a51f2197b8bca540b6c7d7b2b31bafefdc23c701d49ec430efb5d83643754e WatchSource:0}: Error finding container b0a51f2197b8bca540b6c7d7b2b31bafefdc23c701d49ec430efb5d83643754e: Status 404 returned error can't find the container with id b0a51f2197b8bca540b6c7d7b2b31bafefdc23c701d49ec430efb5d83643754e
Jul 17 13:28:03.484618 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:03.484545    2565 generic.go:356] "Generic (PLEG): container finished" podID="19338a8c-78db-48e3-8668-796209c087c1" containerID="efc22da9ce13198e90a76da181dba23230a75745a7af4ab054d0f84adac2f717" exitCode=6
Jul 17 13:28:03.484771 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:03.484639    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb" event={"ID":"19338a8c-78db-48e3-8668-796209c087c1","Type":"ContainerDied","Data":"efc22da9ce13198e90a76da181dba23230a75745a7af4ab054d0f84adac2f717"}
Jul 17 13:28:03.484771 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:03.484704    2565 scope.go:117] "RemoveContainer" containerID="67301fb4cc224b00d7b69d5d4043bf07438501cbd11d0033ee978ff05eb1a1b3"
Jul 17 13:28:03.484870 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:03.484849    2565 scope.go:117] "RemoveContainer" containerID="efc22da9ce13198e90a76da181dba23230a75745a7af4ab054d0f84adac2f717"
Jul 17 13:28:03.485126 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:28:03.485107    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb_odh-ai-gateway-infra(19338a8c-78db-48e3-8668-796209c087c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb" podUID="19338a8c-78db-48e3-8668-796209c087c1"
Jul 17 13:28:03.486382 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:03.486358    2565 generic.go:356] "Generic (PLEG): container finished" podID="72294120-389e-4270-bd7c-1ca994800275" containerID="8249cd40cbf187c557e7c95181e89a1558991e5df86c7a355d4a4e0fcce13832" exitCode=6
Jul 17 13:28:03.486491 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:03.486439    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8" event={"ID":"72294120-389e-4270-bd7c-1ca994800275","Type":"ContainerDied","Data":"8249cd40cbf187c557e7c95181e89a1558991e5df86c7a355d4a4e0fcce13832"}
Jul 17 13:28:03.486682 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:03.486666    2565 scope.go:117] "RemoveContainer" containerID="8249cd40cbf187c557e7c95181e89a1558991e5df86c7a355d4a4e0fcce13832"
Jul 17 13:28:03.487746 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:03.487728    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx" event={"ID":"4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a","Type":"ContainerStarted","Data":"b0a51f2197b8bca540b6c7d7b2b31bafefdc23c701d49ec430efb5d83643754e"}
Jul 17 13:28:04.495826 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:04.495722    2565 generic.go:356] "Generic (PLEG): container finished" podID="4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a" containerID="00513b2987bfbec2bf8c3bc025f918c8068c7f7eba799d4991d1e5a83f321cab" exitCode=6
Jul 17 13:28:04.495826 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:04.495771    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx" event={"ID":"4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a","Type":"ContainerDied","Data":"00513b2987bfbec2bf8c3bc025f918c8068c7f7eba799d4991d1e5a83f321cab"}
Jul 17 13:28:04.496374 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:04.496121    2565 scope.go:117] "RemoveContainer" containerID="00513b2987bfbec2bf8c3bc025f918c8068c7f7eba799d4991d1e5a83f321cab"
Jul 17 13:28:04.498575 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:04.498263    2565 scope.go:117] "RemoveContainer" containerID="efc22da9ce13198e90a76da181dba23230a75745a7af4ab054d0f84adac2f717"
Jul 17 13:28:04.498575 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:28:04.498473    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb_odh-ai-gateway-infra(19338a8c-78db-48e3-8668-796209c087c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb" podUID="19338a8c-78db-48e3-8668-796209c087c1"
Jul 17 13:28:04.502138 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:04.502117    2565 generic.go:356] "Generic (PLEG): container finished" podID="72294120-389e-4270-bd7c-1ca994800275" containerID="f010e89c3951b9b8e6ffcb74db0cb54238eebf86e5b44aa9ab0b4750f71d92d5" exitCode=6
Jul 17 13:28:04.502250 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:04.502155    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8" event={"ID":"72294120-389e-4270-bd7c-1ca994800275","Type":"ContainerDied","Data":"f010e89c3951b9b8e6ffcb74db0cb54238eebf86e5b44aa9ab0b4750f71d92d5"}
Jul 17 13:28:04.502250 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:04.502206    2565 scope.go:117] "RemoveContainer" containerID="8249cd40cbf187c557e7c95181e89a1558991e5df86c7a355d4a4e0fcce13832"
Jul 17 13:28:04.502402 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:04.502387    2565 scope.go:117] "RemoveContainer" containerID="f010e89c3951b9b8e6ffcb74db0cb54238eebf86e5b44aa9ab0b4750f71d92d5"
Jul 17 13:28:04.502595 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:28:04.502577    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8_odh-ai-gateway-infra(72294120-389e-4270-bd7c-1ca994800275)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8" podUID="72294120-389e-4270-bd7c-1ca994800275"
Jul 17 13:28:05.511892 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:05.511803    2565 scope.go:117] "RemoveContainer" containerID="f010e89c3951b9b8e6ffcb74db0cb54238eebf86e5b44aa9ab0b4750f71d92d5"
Jul 17 13:28:05.512322 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:28:05.512078    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8_odh-ai-gateway-infra(72294120-389e-4270-bd7c-1ca994800275)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8" podUID="72294120-389e-4270-bd7c-1ca994800275"
Jul 17 13:28:05.513605 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:05.513577    2565 generic.go:356] "Generic (PLEG): container finished" podID="4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a" containerID="e7c763411b5355401908f2eeda529fc4fd6a62638281043d38946e2a1a5401a5" exitCode=6
Jul 17 13:28:05.513725 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:05.513667    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx" event={"ID":"4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a","Type":"ContainerDied","Data":"e7c763411b5355401908f2eeda529fc4fd6a62638281043d38946e2a1a5401a5"}
Jul 17 13:28:05.513725 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:05.513715    2565 scope.go:117] "RemoveContainer" containerID="00513b2987bfbec2bf8c3bc025f918c8068c7f7eba799d4991d1e5a83f321cab"
Jul 17 13:28:05.513896 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:05.513878    2565 scope.go:117] "RemoveContainer" containerID="e7c763411b5355401908f2eeda529fc4fd6a62638281043d38946e2a1a5401a5"
Jul 17 13:28:05.514136 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:28:05.514115    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx_odh-ai-gateway-infra(4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx" podUID="4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a"
Jul 17 13:28:06.522574 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:06.522546    2565 scope.go:117] "RemoveContainer" containerID="e7c763411b5355401908f2eeda529fc4fd6a62638281043d38946e2a1a5401a5"
Jul 17 13:28:06.522959 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:28:06.522770    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx_odh-ai-gateway-infra(4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx" podUID="4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a"
Jul 17 13:28:13.397503 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:13.397472    2565 scope.go:117] "RemoveContainer" containerID="e7e1e01f14c145fb35614165f36897ec5615be09320a866192c8145a55b2c32a"
Jul 17 13:28:14.579853 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:14.579819    2565 generic.go:356] "Generic (PLEG): container finished" podID="64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25" containerID="4faee588f4575c6f42ade7190dad2c58cfafdfc77c439267b09f16a3f1663666" exitCode=6
Jul 17 13:28:14.580318 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:14.579888    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx" event={"ID":"64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25","Type":"ContainerDied","Data":"4faee588f4575c6f42ade7190dad2c58cfafdfc77c439267b09f16a3f1663666"}
Jul 17 13:28:14.580318 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:14.579947    2565 scope.go:117] "RemoveContainer" containerID="e7e1e01f14c145fb35614165f36897ec5615be09320a866192c8145a55b2c32a"
Jul 17 13:28:14.580318 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:14.580256    2565 scope.go:117] "RemoveContainer" containerID="4faee588f4575c6f42ade7190dad2c58cfafdfc77c439267b09f16a3f1663666"
Jul 17 13:28:14.580535 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:28:14.580504    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx_odh-ai-gateway-infra(64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx" podUID="64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25"
Jul 17 13:28:15.611173 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:15.611138    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx"]
Jul 17 13:28:15.746236 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:15.746211    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx"
Jul 17 13:28:15.829997 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:15.829960    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25-maas-api-service-ca\") pod \"64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25\" (UID: \"64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25\") "
Jul 17 13:28:15.830351 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:15.830327    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25" (UID: "64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:28:15.931645 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:15.931558    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:28:16.400831 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:16.400799    2565 scope.go:117] "RemoveContainer" containerID="efc22da9ce13198e90a76da181dba23230a75745a7af4ab054d0f84adac2f717"
Jul 17 13:28:16.595422 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:16.595389    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx" event={"ID":"64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25","Type":"ContainerDied","Data":"bbad814d552b17900c1172d6abd2f4af1fdc84d3e97466847e7213150e957de9"}
Jul 17 13:28:16.595422 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:16.595407    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx"
Jul 17 13:28:16.595650 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:16.595462    2565 scope.go:117] "RemoveContainer" containerID="4faee588f4575c6f42ade7190dad2c58cfafdfc77c439267b09f16a3f1663666"
Jul 17 13:28:16.614871 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:16.614841    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx"]
Jul 17 13:28:16.623236 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:16.623209    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-lkswx"]
Jul 17 13:28:17.604691 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:17.604658    2565 generic.go:356] "Generic (PLEG): container finished" podID="19338a8c-78db-48e3-8668-796209c087c1" containerID="c6675474814fcd6a600b9c882e1ddb25cda38a9ed094ef4367e73331e8a98c85" exitCode=6
Jul 17 13:28:17.604874 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:17.604735    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb" event={"ID":"19338a8c-78db-48e3-8668-796209c087c1","Type":"ContainerDied","Data":"c6675474814fcd6a600b9c882e1ddb25cda38a9ed094ef4367e73331e8a98c85"}
Jul 17 13:28:17.604874 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:17.604795    2565 scope.go:117] "RemoveContainer" containerID="efc22da9ce13198e90a76da181dba23230a75745a7af4ab054d0f84adac2f717"
Jul 17 13:28:17.605146 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:17.605129    2565 scope.go:117] "RemoveContainer" containerID="c6675474814fcd6a600b9c882e1ddb25cda38a9ed094ef4367e73331e8a98c85"
Jul 17 13:28:17.605387 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:28:17.605369    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb_odh-ai-gateway-infra(19338a8c-78db-48e3-8668-796209c087c1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb" podUID="19338a8c-78db-48e3-8668-796209c087c1"
Jul 17 13:28:18.403301 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:18.403265    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25" path="/var/lib/kubelet/pods/64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25/volumes"
Jul 17 13:28:18.634620 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:18.634585    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb"]
Jul 17 13:28:18.793533 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:18.793504    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb"
Jul 17 13:28:18.962030 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:18.961919    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19338a8c-78db-48e3-8668-796209c087c1-maas-api-service-ca\") pod \"19338a8c-78db-48e3-8668-796209c087c1\" (UID: \"19338a8c-78db-48e3-8668-796209c087c1\") "
Jul 17 13:28:18.962314 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:18.962292    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19338a8c-78db-48e3-8668-796209c087c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "19338a8c-78db-48e3-8668-796209c087c1" (UID: "19338a8c-78db-48e3-8668-796209c087c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:28:19.063214 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:19.063171    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19338a8c-78db-48e3-8668-796209c087c1-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:28:19.628515 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:19.628484    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb"
Jul 17 13:28:19.628941 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:19.628510    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb" event={"ID":"19338a8c-78db-48e3-8668-796209c087c1","Type":"ContainerDied","Data":"9e86adb9520b1648ccecdc66851c333c06fbbb6debcc843bd5443be5829b6f00"}
Jul 17 13:28:19.628941 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:19.628597    2565 scope.go:117] "RemoveContainer" containerID="c6675474814fcd6a600b9c882e1ddb25cda38a9ed094ef4367e73331e8a98c85"
Jul 17 13:28:19.654044 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:19.654007    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb"]
Jul 17 13:28:19.657637 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:19.657611    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-9zwbb"]
Jul 17 13:28:20.397305 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:20.397275    2565 scope.go:117] "RemoveContainer" containerID="f010e89c3951b9b8e6ffcb74db0cb54238eebf86e5b44aa9ab0b4750f71d92d5"
Jul 17 13:28:20.404061 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:20.404036    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19338a8c-78db-48e3-8668-796209c087c1" path="/var/lib/kubelet/pods/19338a8c-78db-48e3-8668-796209c087c1/volumes"
Jul 17 13:28:21.644913 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:21.644879    2565 generic.go:356] "Generic (PLEG): container finished" podID="72294120-389e-4270-bd7c-1ca994800275" containerID="0b0b8ee87d5de288fa9070a8222801678bccf8727efe44dc72a3cdd0580e7814" exitCode=6
Jul 17 13:28:21.645340 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:21.644951    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8" event={"ID":"72294120-389e-4270-bd7c-1ca994800275","Type":"ContainerDied","Data":"0b0b8ee87d5de288fa9070a8222801678bccf8727efe44dc72a3cdd0580e7814"}
Jul 17 13:28:21.645340 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:21.645033    2565 scope.go:117] "RemoveContainer" containerID="f010e89c3951b9b8e6ffcb74db0cb54238eebf86e5b44aa9ab0b4750f71d92d5"
Jul 17 13:28:21.645340 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:21.645326    2565 scope.go:117] "RemoveContainer" containerID="0b0b8ee87d5de288fa9070a8222801678bccf8727efe44dc72a3cdd0580e7814"
Jul 17 13:28:21.645577 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:28:21.645556    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8_odh-ai-gateway-infra(72294120-389e-4270-bd7c-1ca994800275)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8" podUID="72294120-389e-4270-bd7c-1ca994800275"
Jul 17 13:28:22.397033 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:22.396966    2565 scope.go:117] "RemoveContainer" containerID="e7c763411b5355401908f2eeda529fc4fd6a62638281043d38946e2a1a5401a5"
Jul 17 13:28:22.672633 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:22.672543    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8"]
Jul 17 13:28:22.820025 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:22.819969    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8"
Jul 17 13:28:22.999360 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:22.999253    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72294120-389e-4270-bd7c-1ca994800275-maas-api-service-ca\") pod \"72294120-389e-4270-bd7c-1ca994800275\" (UID: \"72294120-389e-4270-bd7c-1ca994800275\") "
Jul 17 13:28:22.999632 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:22.999609    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72294120-389e-4270-bd7c-1ca994800275-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "72294120-389e-4270-bd7c-1ca994800275" (UID: "72294120-389e-4270-bd7c-1ca994800275"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:28:23.100996 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:23.100956    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72294120-389e-4270-bd7c-1ca994800275-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:28:23.660783 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:23.660752    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8"
Jul 17 13:28:23.661021 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:23.660781    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8" event={"ID":"72294120-389e-4270-bd7c-1ca994800275","Type":"ContainerDied","Data":"7e293057e1548c90a70e1d91b271e0eeb413f7c12d3a5465b852a2e6cb32990a"}
Jul 17 13:28:23.661021 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:23.660851    2565 scope.go:117] "RemoveContainer" containerID="0b0b8ee87d5de288fa9070a8222801678bccf8727efe44dc72a3cdd0580e7814"
Jul 17 13:28:23.662870 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:23.662835    2565 generic.go:356] "Generic (PLEG): container finished" podID="4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a" containerID="c3c044a523e3b7cbf306d7a44d5efa27f5c6a6cf6d5900e1b61ece5f1c48e9d3" exitCode=6
Jul 17 13:28:23.662973 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:23.662864    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx" event={"ID":"4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a","Type":"ContainerDied","Data":"c3c044a523e3b7cbf306d7a44d5efa27f5c6a6cf6d5900e1b61ece5f1c48e9d3"}
Jul 17 13:28:23.663298 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:23.663281    2565 scope.go:117] "RemoveContainer" containerID="c3c044a523e3b7cbf306d7a44d5efa27f5c6a6cf6d5900e1b61ece5f1c48e9d3"
Jul 17 13:28:23.663556 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:28:23.663531    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx_odh-ai-gateway-infra(4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx" podUID="4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a"
Jul 17 13:28:23.671439 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:23.671420    2565 scope.go:117] "RemoveContainer" containerID="e7c763411b5355401908f2eeda529fc4fd6a62638281043d38946e2a1a5401a5"
Jul 17 13:28:23.694441 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:23.694418    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8"]
Jul 17 13:28:23.698355 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:23.698329    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-zx8t8"]
Jul 17 13:28:24.409729 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:24.409693    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72294120-389e-4270-bd7c-1ca994800275" path="/var/lib/kubelet/pods/72294120-389e-4270-bd7c-1ca994800275/volumes"
Jul 17 13:28:24.691707 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:24.691613    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx"]
Jul 17 13:28:24.844352 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:24.844327    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx"
Jul 17 13:28:25.021379 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:25.021288    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a-maas-api-service-ca\") pod \"4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a\" (UID: \"4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a\") "
Jul 17 13:28:25.021605 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:25.021584    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a" (UID: "4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:28:25.122837 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:25.122794    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:28:25.679461 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:25.679422    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx" event={"ID":"4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a","Type":"ContainerDied","Data":"b0a51f2197b8bca540b6c7d7b2b31bafefdc23c701d49ec430efb5d83643754e"}
Jul 17 13:28:25.679662 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:25.679473    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx"
Jul 17 13:28:25.679662 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:25.679496    2565 scope.go:117] "RemoveContainer" containerID="c3c044a523e3b7cbf306d7a44d5efa27f5c6a6cf6d5900e1b61ece5f1c48e9d3"
Jul 17 13:28:25.702939 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:25.702900    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx"]
Jul 17 13:28:25.708418 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:25.708391    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-qlczx"]
Jul 17 13:28:26.403963 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:26.403929    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a" path="/var/lib/kubelet/pods/4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a/volumes"
Jul 17 13:28:34.318296 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:34.318264    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:28:34.414851 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:34.414807    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:28:49.101847 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.101811    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4"]
Jul 17 13:28:49.102320 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102273    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19338a8c-78db-48e3-8668-796209c087c1" containerName="revoke-keys"
Jul 17 13:28:49.102320 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102284    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="19338a8c-78db-48e3-8668-796209c087c1" containerName="revoke-keys"
Jul 17 13:28:49.102320 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102292    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72294120-389e-4270-bd7c-1ca994800275" containerName="revoke-keys"
Jul 17 13:28:49.102320 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102297    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="72294120-389e-4270-bd7c-1ca994800275" containerName="revoke-keys"
Jul 17 13:28:49.102320 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102306    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25" containerName="revoke-keys"
Jul 17 13:28:49.102320 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102311    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25" containerName="revoke-keys"
Jul 17 13:28:49.102320 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102317    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a" containerName="revoke-keys"
Jul 17 13:28:49.102320 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102322    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102331    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19338a8c-78db-48e3-8668-796209c087c1" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102338    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="19338a8c-78db-48e3-8668-796209c087c1" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102344    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102349    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102356    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72294120-389e-4270-bd7c-1ca994800275" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102361    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="72294120-389e-4270-bd7c-1ca994800275" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102370    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19338a8c-78db-48e3-8668-796209c087c1" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102375    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="19338a8c-78db-48e3-8668-796209c087c1" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102384    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72294120-389e-4270-bd7c-1ca994800275" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102389    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="72294120-389e-4270-bd7c-1ca994800275" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102397    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102401    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102411    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102415    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102477    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102484    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="72294120-389e-4270-bd7c-1ca994800275" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102489    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102496    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102502    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="19338a8c-78db-48e3-8668-796209c087c1" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102509    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102513    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="72294120-389e-4270-bd7c-1ca994800275" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102518    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="19338a8c-78db-48e3-8668-796209c087c1" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102523    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="19338a8c-78db-48e3-8668-796209c087c1" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102528    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="72294120-389e-4270-bd7c-1ca994800275" containerName="revoke-keys"
Jul 17 13:28:49.102576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.102534    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="64aa95a7-ea2f-4c38-a9c0-4a89c0ae6b25" containerName="revoke-keys"
Jul 17 13:28:49.107650 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.107630    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4"
Jul 17 13:28:49.110649 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.110505    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:28:49.113722 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.113635    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4"]
Jul 17 13:28:49.240743 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.240707    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc09458-b10f-4744-ac4d-b01b4af16f2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4\" (UID: \"6cc09458-b10f-4744-ac4d-b01b4af16f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4"
Jul 17 13:28:49.341908 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.341871    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc09458-b10f-4744-ac4d-b01b4af16f2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4\" (UID: \"6cc09458-b10f-4744-ac4d-b01b4af16f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4"
Jul 17 13:28:49.342537 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.342516    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc09458-b10f-4744-ac4d-b01b4af16f2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4\" (UID: \"6cc09458-b10f-4744-ac4d-b01b4af16f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4"
Jul 17 13:28:49.421931 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.421841    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4"
Jul 17 13:28:49.573246 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.573206    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4"]
Jul 17 13:28:49.574850 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:28:49.574824    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc09458_b10f_4744_ac4d_b01b4af16f2c.slice/crio-be5d0a37bd4ba8c45b0a8c86f05636955d2826b4f93da2e8938f0f7f07bc9e53 WatchSource:0}: Error finding container be5d0a37bd4ba8c45b0a8c86f05636955d2826b4f93da2e8938f0f7f07bc9e53: Status 404 returned error can't find the container with id be5d0a37bd4ba8c45b0a8c86f05636955d2826b4f93da2e8938f0f7f07bc9e53
Jul 17 13:28:49.835921 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:49.835885    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4" event={"ID":"6cc09458-b10f-4744-ac4d-b01b4af16f2c","Type":"ContainerStarted","Data":"be5d0a37bd4ba8c45b0a8c86f05636955d2826b4f93da2e8938f0f7f07bc9e53"}
Jul 17 13:28:50.377456 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:50.377413    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr"]
Jul 17 13:28:50.377945 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:50.377931    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a" containerName="revoke-keys"
Jul 17 13:28:50.377945 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:50.377946    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a" containerName="revoke-keys"
Jul 17 13:28:50.378045 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:50.378027    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f6497d8-4fd6-479b-ba90-ad6d3bb19c6a" containerName="revoke-keys"
Jul 17 13:28:50.381223 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:50.381208    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr"
Jul 17 13:28:50.387748 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:50.387621    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr"]
Jul 17 13:28:50.454643 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:50.454611    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/847bd07a-fd90-4aa4-8fc8-2ad3e2142046-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr\" (UID: \"847bd07a-fd90-4aa4-8fc8-2ad3e2142046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr"
Jul 17 13:28:50.555779 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:50.555737    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/847bd07a-fd90-4aa4-8fc8-2ad3e2142046-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr\" (UID: \"847bd07a-fd90-4aa4-8fc8-2ad3e2142046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr"
Jul 17 13:28:50.556397 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:50.556375    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/847bd07a-fd90-4aa4-8fc8-2ad3e2142046-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr\" (UID: \"847bd07a-fd90-4aa4-8fc8-2ad3e2142046\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr"
Jul 17 13:28:50.694328 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:50.694237    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr"
Jul 17 13:28:50.845863 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:50.845775    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr"]
Jul 17 13:28:50.848661 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:28:50.848633    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod847bd07a_fd90_4aa4_8fc8_2ad3e2142046.slice/crio-383487e5e21e4f4dbfa56fdc92c780fb8c34844901515f3e27f1bc7bab58ed60 WatchSource:0}: Error finding container 383487e5e21e4f4dbfa56fdc92c780fb8c34844901515f3e27f1bc7bab58ed60: Status 404 returned error can't find the container with id 383487e5e21e4f4dbfa56fdc92c780fb8c34844901515f3e27f1bc7bab58ed60
Jul 17 13:28:50.850762 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:50.850732    2565 generic.go:356] "Generic (PLEG): container finished" podID="6cc09458-b10f-4744-ac4d-b01b4af16f2c" containerID="9464fd7e04c4b23b97f90b4b5eb47ac5f8cf36ea6087a567310698a496904f33" exitCode=6
Jul 17 13:28:50.850872 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:50.850796    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4" event={"ID":"6cc09458-b10f-4744-ac4d-b01b4af16f2c","Type":"ContainerDied","Data":"9464fd7e04c4b23b97f90b4b5eb47ac5f8cf36ea6087a567310698a496904f33"}
Jul 17 13:28:50.851070 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:50.851054    2565 scope.go:117] "RemoveContainer" containerID="9464fd7e04c4b23b97f90b4b5eb47ac5f8cf36ea6087a567310698a496904f33"
Jul 17 13:28:51.859190 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:51.859160    2565 generic.go:356] "Generic (PLEG): container finished" podID="6cc09458-b10f-4744-ac4d-b01b4af16f2c" containerID="238bffd673fda65e87407df12b35b715e4138fe5f08e367ee3afcd2bf01f76b8" exitCode=6
Jul 17 13:28:51.859631 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:51.859227    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4" event={"ID":"6cc09458-b10f-4744-ac4d-b01b4af16f2c","Type":"ContainerDied","Data":"238bffd673fda65e87407df12b35b715e4138fe5f08e367ee3afcd2bf01f76b8"}
Jul 17 13:28:51.859631 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:51.859287    2565 scope.go:117] "RemoveContainer" containerID="9464fd7e04c4b23b97f90b4b5eb47ac5f8cf36ea6087a567310698a496904f33"
Jul 17 13:28:51.859631 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:51.859484    2565 scope.go:117] "RemoveContainer" containerID="238bffd673fda65e87407df12b35b715e4138fe5f08e367ee3afcd2bf01f76b8"
Jul 17 13:28:51.859764 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:28:51.859733    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4_odh-ai-gateway-infra(6cc09458-b10f-4744-ac4d-b01b4af16f2c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4" podUID="6cc09458-b10f-4744-ac4d-b01b4af16f2c"
Jul 17 13:28:51.861029 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:51.861009    2565 generic.go:356] "Generic (PLEG): container finished" podID="847bd07a-fd90-4aa4-8fc8-2ad3e2142046" containerID="e8de892a8dc1347814ed1b88273c907566350f2e679bf6afc8f672c0b9ddc998" exitCode=6
Jul 17 13:28:51.861147 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:51.861127    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr" event={"ID":"847bd07a-fd90-4aa4-8fc8-2ad3e2142046","Type":"ContainerDied","Data":"e8de892a8dc1347814ed1b88273c907566350f2e679bf6afc8f672c0b9ddc998"}
Jul 17 13:28:51.861196 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:51.861175    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr" event={"ID":"847bd07a-fd90-4aa4-8fc8-2ad3e2142046","Type":"ContainerStarted","Data":"383487e5e21e4f4dbfa56fdc92c780fb8c34844901515f3e27f1bc7bab58ed60"}
Jul 17 13:28:51.861229 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:51.861212    2565 scope.go:117] "RemoveContainer" containerID="e8de892a8dc1347814ed1b88273c907566350f2e679bf6afc8f672c0b9ddc998"
Jul 17 13:28:52.869515 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:52.869477    2565 scope.go:117] "RemoveContainer" containerID="238bffd673fda65e87407df12b35b715e4138fe5f08e367ee3afcd2bf01f76b8"
Jul 17 13:28:52.869949 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:28:52.869711    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4_odh-ai-gateway-infra(6cc09458-b10f-4744-ac4d-b01b4af16f2c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4" podUID="6cc09458-b10f-4744-ac4d-b01b4af16f2c"
Jul 17 13:28:52.871227 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:52.871201    2565 generic.go:356] "Generic (PLEG): container finished" podID="847bd07a-fd90-4aa4-8fc8-2ad3e2142046" containerID="3900b3308f5e6fc08334bfe1527641a36c6cb88106da646a185cb7e407db3c16" exitCode=6
Jul 17 13:28:52.871336 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:52.871289    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr" event={"ID":"847bd07a-fd90-4aa4-8fc8-2ad3e2142046","Type":"ContainerDied","Data":"3900b3308f5e6fc08334bfe1527641a36c6cb88106da646a185cb7e407db3c16"}
Jul 17 13:28:52.871382 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:52.871345    2565 scope.go:117] "RemoveContainer" containerID="e8de892a8dc1347814ed1b88273c907566350f2e679bf6afc8f672c0b9ddc998"
Jul 17 13:28:52.871458 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:52.871442    2565 scope.go:117] "RemoveContainer" containerID="3900b3308f5e6fc08334bfe1527641a36c6cb88106da646a185cb7e407db3c16"
Jul 17 13:28:52.871672 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:28:52.871653    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr_odh-ai-gateway-infra(847bd07a-fd90-4aa4-8fc8-2ad3e2142046)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr" podUID="847bd07a-fd90-4aa4-8fc8-2ad3e2142046"
Jul 17 13:28:53.879476 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:28:53.879448    2565 scope.go:117] "RemoveContainer" containerID="3900b3308f5e6fc08334bfe1527641a36c6cb88106da646a185cb7e407db3c16"
Jul 17 13:28:53.879864 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:28:53.879660    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr_odh-ai-gateway-infra(847bd07a-fd90-4aa4-8fc8-2ad3e2142046)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr" podUID="847bd07a-fd90-4aa4-8fc8-2ad3e2142046"
Jul 17 13:29:01.627766 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:01.627725    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf"]
Jul 17 13:29:01.632430 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:01.632411    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf"
Jul 17 13:29:01.638582 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:01.638552    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf"]
Jul 17 13:29:01.763558 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:01.763513    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9463ee-1100-445a-8744-0584bb5f138f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf\" (UID: \"ce9463ee-1100-445a-8744-0584bb5f138f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf"
Jul 17 13:29:01.864870 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:01.864829    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9463ee-1100-445a-8744-0584bb5f138f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf\" (UID: \"ce9463ee-1100-445a-8744-0584bb5f138f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf"
Jul 17 13:29:01.865509 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:01.865491    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9463ee-1100-445a-8744-0584bb5f138f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf\" (UID: \"ce9463ee-1100-445a-8744-0584bb5f138f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf"
Jul 17 13:29:01.945856 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:01.945768    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf"
Jul 17 13:29:02.093130 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:29:02.093090    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9463ee_1100_445a_8744_0584bb5f138f.slice/crio-3fdab60641471637b8e1ff020c7232b713c4f9952bda732e464cc297011b9824 WatchSource:0}: Error finding container 3fdab60641471637b8e1ff020c7232b713c4f9952bda732e464cc297011b9824: Status 404 returned error can't find the container with id 3fdab60641471637b8e1ff020c7232b713c4f9952bda732e464cc297011b9824
Jul 17 13:29:02.093327 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:02.093301    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf"]
Jul 17 13:29:02.935147 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:02.935111    2565 generic.go:356] "Generic (PLEG): container finished" podID="ce9463ee-1100-445a-8744-0584bb5f138f" containerID="e7858e4780c9d319ecb5b0c5408326b81bc06dd4c029a6d5370948b346c4dff0" exitCode=6
Jul 17 13:29:02.935630 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:02.935207    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf" event={"ID":"ce9463ee-1100-445a-8744-0584bb5f138f","Type":"ContainerDied","Data":"e7858e4780c9d319ecb5b0c5408326b81bc06dd4c029a6d5370948b346c4dff0"}
Jul 17 13:29:02.935630 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:02.935262    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf" event={"ID":"ce9463ee-1100-445a-8744-0584bb5f138f","Type":"ContainerStarted","Data":"3fdab60641471637b8e1ff020c7232b713c4f9952bda732e464cc297011b9824"}
Jul 17 13:29:02.935630 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:02.935352    2565 scope.go:117] "RemoveContainer" containerID="e7858e4780c9d319ecb5b0c5408326b81bc06dd4c029a6d5370948b346c4dff0"
Jul 17 13:29:02.992635 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:02.992602    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8"]
Jul 17 13:29:02.997531 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:02.997512    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8"
Jul 17 13:29:03.007924 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:03.006326    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8"]
Jul 17 13:29:03.077742 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:03.077647    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e34a350-9601-41aa-b744-e43b9fef5d59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8\" (UID: \"5e34a350-9601-41aa-b744-e43b9fef5d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8"
Jul 17 13:29:03.179205 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:03.179169    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e34a350-9601-41aa-b744-e43b9fef5d59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8\" (UID: \"5e34a350-9601-41aa-b744-e43b9fef5d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8"
Jul 17 13:29:03.179784 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:03.179764    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e34a350-9601-41aa-b744-e43b9fef5d59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8\" (UID: \"5e34a350-9601-41aa-b744-e43b9fef5d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8"
Jul 17 13:29:03.312084 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:03.312053    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8"
Jul 17 13:29:03.460824 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:03.460800    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8"]
Jul 17 13:29:03.462200 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:29:03.462171    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e34a350_9601_41aa_b744_e43b9fef5d59.slice/crio-cd79097679d177283b0913c308a67f83d0bc612f8507d6c167cdd6873d31cc29 WatchSource:0}: Error finding container cd79097679d177283b0913c308a67f83d0bc612f8507d6c167cdd6873d31cc29: Status 404 returned error can't find the container with id cd79097679d177283b0913c308a67f83d0bc612f8507d6c167cdd6873d31cc29
Jul 17 13:29:03.943139 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:03.943020    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8" event={"ID":"5e34a350-9601-41aa-b744-e43b9fef5d59","Type":"ContainerStarted","Data":"d6e146a475d9f8c2f0311778a0d3129b88ee49882dfd6bb9553c7e425b1d4318"}
Jul 17 13:29:03.943139 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:03.943089    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8" event={"ID":"5e34a350-9601-41aa-b744-e43b9fef5d59","Type":"ContainerStarted","Data":"cd79097679d177283b0913c308a67f83d0bc612f8507d6c167cdd6873d31cc29"}
Jul 17 13:29:03.944814 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:03.944793    2565 generic.go:356] "Generic (PLEG): container finished" podID="ce9463ee-1100-445a-8744-0584bb5f138f" containerID="52f715f571dc9c154724b85e33cdb6c246b2000785ef1959ac4ef228d8863ac5" exitCode=6
Jul 17 13:29:03.944931 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:03.944878    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf" event={"ID":"ce9463ee-1100-445a-8744-0584bb5f138f","Type":"ContainerDied","Data":"52f715f571dc9c154724b85e33cdb6c246b2000785ef1959ac4ef228d8863ac5"}
Jul 17 13:29:03.945007 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:03.944932    2565 scope.go:117] "RemoveContainer" containerID="e7858e4780c9d319ecb5b0c5408326b81bc06dd4c029a6d5370948b346c4dff0"
Jul 17 13:29:03.945070 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:03.945057    2565 scope.go:117] "RemoveContainer" containerID="52f715f571dc9c154724b85e33cdb6c246b2000785ef1959ac4ef228d8863ac5"
Jul 17 13:29:03.945277 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:29:03.945258    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf_odh-ai-gateway-infra(ce9463ee-1100-445a-8744-0584bb5f138f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf" podUID="ce9463ee-1100-445a-8744-0584bb5f138f"
Jul 17 13:29:03.961275 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:03.961228    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8" podStartSLOduration=1.961211559 podStartE2EDuration="1.961211559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:29:03.960438097 +0000 UTC m=+3988.091822758" watchObservedRunningTime="2026-07-17 13:29:03.961211559 +0000 UTC m=+3988.092596219"
Jul 17 13:29:04.952798 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:04.952756    2565 generic.go:356] "Generic (PLEG): container finished" podID="5e34a350-9601-41aa-b744-e43b9fef5d59" containerID="d6e146a475d9f8c2f0311778a0d3129b88ee49882dfd6bb9553c7e425b1d4318" exitCode=6
Jul 17 13:29:04.953306 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:04.952832    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8" event={"ID":"5e34a350-9601-41aa-b744-e43b9fef5d59","Type":"ContainerDied","Data":"d6e146a475d9f8c2f0311778a0d3129b88ee49882dfd6bb9553c7e425b1d4318"}
Jul 17 13:29:04.953306 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:04.953072    2565 scope.go:117] "RemoveContainer" containerID="d6e146a475d9f8c2f0311778a0d3129b88ee49882dfd6bb9553c7e425b1d4318"
Jul 17 13:29:04.954894 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:04.954877    2565 scope.go:117] "RemoveContainer" containerID="52f715f571dc9c154724b85e33cdb6c246b2000785ef1959ac4ef228d8863ac5"
Jul 17 13:29:04.955164 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:29:04.955128    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf_odh-ai-gateway-infra(ce9463ee-1100-445a-8744-0584bb5f138f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf" podUID="ce9463ee-1100-445a-8744-0584bb5f138f"
Jul 17 13:29:05.403550 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:05.397147    2565 scope.go:117] "RemoveContainer" containerID="3900b3308f5e6fc08334bfe1527641a36c6cb88106da646a185cb7e407db3c16"
Jul 17 13:29:05.963253 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:05.963151    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr" event={"ID":"847bd07a-fd90-4aa4-8fc8-2ad3e2142046","Type":"ContainerStarted","Data":"e7b1ae84b11ed60c8b225fab1b5a74fdc97373bc4e54c674e403cc401dc229f0"}
Jul 17 13:29:05.965044 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:05.965014    2565 generic.go:356] "Generic (PLEG): container finished" podID="5e34a350-9601-41aa-b744-e43b9fef5d59" containerID="af776563f25ac48ca4cbb3fa653f56010c2cd24c882dbe1cade4b620c511dd11" exitCode=6
Jul 17 13:29:05.965119 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:05.965061    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8" event={"ID":"5e34a350-9601-41aa-b744-e43b9fef5d59","Type":"ContainerDied","Data":"af776563f25ac48ca4cbb3fa653f56010c2cd24c882dbe1cade4b620c511dd11"}
Jul 17 13:29:05.965119 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:05.965111    2565 scope.go:117] "RemoveContainer" containerID="d6e146a475d9f8c2f0311778a0d3129b88ee49882dfd6bb9553c7e425b1d4318"
Jul 17 13:29:05.965331 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:05.965318    2565 scope.go:117] "RemoveContainer" containerID="af776563f25ac48ca4cbb3fa653f56010c2cd24c882dbe1cade4b620c511dd11"
Jul 17 13:29:05.965549 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:29:05.965530    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8_odh-ai-gateway-infra(5e34a350-9601-41aa-b744-e43b9fef5d59)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8" podUID="5e34a350-9601-41aa-b744-e43b9fef5d59"
Jul 17 13:29:05.982195 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:05.982150    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr" podStartSLOduration=15.982135584 podStartE2EDuration="15.982135584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:29:05.981025662 +0000 UTC m=+3990.112410344" watchObservedRunningTime="2026-07-17 13:29:05.982135584 +0000 UTC m=+3990.113520244"
Jul 17 13:29:06.973858 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:06.973828    2565 scope.go:117] "RemoveContainer" containerID="af776563f25ac48ca4cbb3fa653f56010c2cd24c882dbe1cade4b620c511dd11"
Jul 17 13:29:06.974323 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:29:06.974085    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8_odh-ai-gateway-infra(5e34a350-9601-41aa-b744-e43b9fef5d59)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8" podUID="5e34a350-9601-41aa-b744-e43b9fef5d59"
Jul 17 13:29:06.975586 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:06.975535    2565 generic.go:356] "Generic (PLEG): container finished" podID="847bd07a-fd90-4aa4-8fc8-2ad3e2142046" containerID="e7b1ae84b11ed60c8b225fab1b5a74fdc97373bc4e54c674e403cc401dc229f0" exitCode=6
Jul 17 13:29:06.975704 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:06.975606    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr" event={"ID":"847bd07a-fd90-4aa4-8fc8-2ad3e2142046","Type":"ContainerDied","Data":"e7b1ae84b11ed60c8b225fab1b5a74fdc97373bc4e54c674e403cc401dc229f0"}
Jul 17 13:29:06.975704 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:06.975661    2565 scope.go:117] "RemoveContainer" containerID="3900b3308f5e6fc08334bfe1527641a36c6cb88106da646a185cb7e407db3c16"
Jul 17 13:29:06.975930 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:06.975914    2565 scope.go:117] "RemoveContainer" containerID="e7b1ae84b11ed60c8b225fab1b5a74fdc97373bc4e54c674e403cc401dc229f0"
Jul 17 13:29:06.976173 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:29:06.976152    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr_odh-ai-gateway-infra(847bd07a-fd90-4aa4-8fc8-2ad3e2142046)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr" podUID="847bd07a-fd90-4aa4-8fc8-2ad3e2142046"
Jul 17 13:29:06.993709 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:06.993674    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr"]
Jul 17 13:29:07.397154 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:07.397117    2565 scope.go:117] "RemoveContainer" containerID="238bffd673fda65e87407df12b35b715e4138fe5f08e367ee3afcd2bf01f76b8"
Jul 17 13:29:07.984538 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:07.984450    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4" event={"ID":"6cc09458-b10f-4744-ac4d-b01b4af16f2c","Type":"ContainerStarted","Data":"e9c3b28072a1cbe6912b7e8cd0b8bc8a856811bb6c0f9f147cceb96a2e22806f"}
Jul 17 13:29:08.002686 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:08.002636    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4" podStartSLOduration=19.002618274 podStartE2EDuration="19.002618274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:29:08.001935526 +0000 UTC m=+3992.133320207" watchObservedRunningTime="2026-07-17 13:29:08.002618274 +0000 UTC m=+3992.134002936"
Jul 17 13:29:08.138363 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:08.138338    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr"
Jul 17 13:29:08.235133 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:08.235058    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/847bd07a-fd90-4aa4-8fc8-2ad3e2142046-maas-api-service-ca\") pod \"847bd07a-fd90-4aa4-8fc8-2ad3e2142046\" (UID: \"847bd07a-fd90-4aa4-8fc8-2ad3e2142046\") "
Jul 17 13:29:08.235389 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:08.235368    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/847bd07a-fd90-4aa4-8fc8-2ad3e2142046-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "847bd07a-fd90-4aa4-8fc8-2ad3e2142046" (UID: "847bd07a-fd90-4aa4-8fc8-2ad3e2142046"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:29:08.336492 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:08.336454    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/847bd07a-fd90-4aa4-8fc8-2ad3e2142046-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:29:08.993734 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:08.993693    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr" event={"ID":"847bd07a-fd90-4aa4-8fc8-2ad3e2142046","Type":"ContainerDied","Data":"383487e5e21e4f4dbfa56fdc92c780fb8c34844901515f3e27f1bc7bab58ed60"}
Jul 17 13:29:08.994239 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:08.993791    2565 scope.go:117] "RemoveContainer" containerID="e7b1ae84b11ed60c8b225fab1b5a74fdc97373bc4e54c674e403cc401dc229f0"
Jul 17 13:29:08.994239 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:08.993707    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr"
Jul 17 13:29:08.995925 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:08.995900    2565 generic.go:356] "Generic (PLEG): container finished" podID="6cc09458-b10f-4744-ac4d-b01b4af16f2c" containerID="e9c3b28072a1cbe6912b7e8cd0b8bc8a856811bb6c0f9f147cceb96a2e22806f" exitCode=6
Jul 17 13:29:08.996146 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:08.995936    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4" event={"ID":"6cc09458-b10f-4744-ac4d-b01b4af16f2c","Type":"ContainerDied","Data":"e9c3b28072a1cbe6912b7e8cd0b8bc8a856811bb6c0f9f147cceb96a2e22806f"}
Jul 17 13:29:08.996304 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:08.996290    2565 scope.go:117] "RemoveContainer" containerID="e9c3b28072a1cbe6912b7e8cd0b8bc8a856811bb6c0f9f147cceb96a2e22806f"
Jul 17 13:29:08.996594 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:29:08.996549    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4_odh-ai-gateway-infra(6cc09458-b10f-4744-ac4d-b01b4af16f2c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4" podUID="6cc09458-b10f-4744-ac4d-b01b4af16f2c"
Jul 17 13:29:09.003704 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:09.003684    2565 scope.go:117] "RemoveContainer" containerID="238bffd673fda65e87407df12b35b715e4138fe5f08e367ee3afcd2bf01f76b8"
Jul 17 13:29:09.015389 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:09.015364    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr"]
Jul 17 13:29:09.019565 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:09.019542    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4"]
Jul 17 13:29:09.021970 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:09.021943    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-cfkwr"]
Jul 17 13:29:10.153402 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:10.153377    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4"
Jul 17 13:29:10.257898 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:10.257795    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc09458-b10f-4744-ac4d-b01b4af16f2c-maas-api-service-ca\") pod \"6cc09458-b10f-4744-ac4d-b01b4af16f2c\" (UID: \"6cc09458-b10f-4744-ac4d-b01b4af16f2c\") "
Jul 17 13:29:10.258214 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:10.258189    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc09458-b10f-4744-ac4d-b01b4af16f2c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6cc09458-b10f-4744-ac4d-b01b4af16f2c" (UID: "6cc09458-b10f-4744-ac4d-b01b4af16f2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:29:10.359193 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:10.359153    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc09458-b10f-4744-ac4d-b01b4af16f2c-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:29:10.402929 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:10.402898    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="847bd07a-fd90-4aa4-8fc8-2ad3e2142046" path="/var/lib/kubelet/pods/847bd07a-fd90-4aa4-8fc8-2ad3e2142046/volumes"
Jul 17 13:29:11.012967 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:11.012931    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4" event={"ID":"6cc09458-b10f-4744-ac4d-b01b4af16f2c","Type":"ContainerDied","Data":"be5d0a37bd4ba8c45b0a8c86f05636955d2826b4f93da2e8938f0f7f07bc9e53"}
Jul 17 13:29:11.012967 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:11.012950    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4"
Jul 17 13:29:11.013250 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:11.013029    2565 scope.go:117] "RemoveContainer" containerID="e9c3b28072a1cbe6912b7e8cd0b8bc8a856811bb6c0f9f147cceb96a2e22806f"
Jul 17 13:29:11.032738 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:11.032708    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4"]
Jul 17 13:29:11.037605 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:11.037578    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-kxbr4"]
Jul 17 13:29:12.403340 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:12.403305    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cc09458-b10f-4744-ac4d-b01b4af16f2c" path="/var/lib/kubelet/pods/6cc09458-b10f-4744-ac4d-b01b4af16f2c/volumes"
Jul 17 13:29:15.984344 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:15.984309    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:29:16.044470 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:16.044427    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:29:16.400566 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:16.400533    2565 scope.go:117] "RemoveContainer" containerID="52f715f571dc9c154724b85e33cdb6c246b2000785ef1959ac4ef228d8863ac5"
Jul 17 13:29:17.053676 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:17.053640    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf" event={"ID":"ce9463ee-1100-445a-8744-0584bb5f138f","Type":"ContainerStarted","Data":"7b2002efc0019503587490a84effea6f4111320d773f0155eef8a6a864eab7d9"}
Jul 17 13:29:17.073821 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:17.073770    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf" podStartSLOduration=16.07375511 podStartE2EDuration="16.07375511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:29:17.072073283 +0000 UTC m=+4001.203457965" watchObservedRunningTime="2026-07-17 13:29:17.07375511 +0000 UTC m=+4001.205139770"
Jul 17 13:29:18.061671 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:18.061631    2565 generic.go:356] "Generic (PLEG): container finished" podID="ce9463ee-1100-445a-8744-0584bb5f138f" containerID="7b2002efc0019503587490a84effea6f4111320d773f0155eef8a6a864eab7d9" exitCode=6
Jul 17 13:29:18.062146 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:18.061712    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf" event={"ID":"ce9463ee-1100-445a-8744-0584bb5f138f","Type":"ContainerDied","Data":"7b2002efc0019503587490a84effea6f4111320d773f0155eef8a6a864eab7d9"}
Jul 17 13:29:18.062146 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:18.061777    2565 scope.go:117] "RemoveContainer" containerID="52f715f571dc9c154724b85e33cdb6c246b2000785ef1959ac4ef228d8863ac5"
Jul 17 13:29:18.062146 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:18.062116    2565 scope.go:117] "RemoveContainer" containerID="7b2002efc0019503587490a84effea6f4111320d773f0155eef8a6a864eab7d9"
Jul 17 13:29:18.062376 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:29:18.062358    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf_odh-ai-gateway-infra(ce9463ee-1100-445a-8744-0584bb5f138f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf" podUID="ce9463ee-1100-445a-8744-0584bb5f138f"
Jul 17 13:29:18.090759 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:18.090732    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf"]
Jul 17 13:29:18.396881 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:18.396799    2565 scope.go:117] "RemoveContainer" containerID="af776563f25ac48ca4cbb3fa653f56010c2cd24c882dbe1cade4b620c511dd11"
Jul 17 13:29:19.070174 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:19.070139    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8" event={"ID":"5e34a350-9601-41aa-b744-e43b9fef5d59","Type":"ContainerStarted","Data":"3422e9aceec1c4508d5494e939281f24be27498b415a0c4f9a87888ab86cc810"}
Jul 17 13:29:19.217270 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:19.217244    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf"
Jul 17 13:29:19.237341 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:19.237314    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9463ee-1100-445a-8744-0584bb5f138f-maas-api-service-ca\") pod \"ce9463ee-1100-445a-8744-0584bb5f138f\" (UID: \"ce9463ee-1100-445a-8744-0584bb5f138f\") "
Jul 17 13:29:19.237754 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:19.237726    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce9463ee-1100-445a-8744-0584bb5f138f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ce9463ee-1100-445a-8744-0584bb5f138f" (UID: "ce9463ee-1100-445a-8744-0584bb5f138f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:29:19.338326 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:19.338243    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9463ee-1100-445a-8744-0584bb5f138f-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:29:20.079862 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:20.079831    2565 generic.go:356] "Generic (PLEG): container finished" podID="5e34a350-9601-41aa-b744-e43b9fef5d59" containerID="3422e9aceec1c4508d5494e939281f24be27498b415a0c4f9a87888ab86cc810" exitCode=6
Jul 17 13:29:20.080350 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:20.079909    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8" event={"ID":"5e34a350-9601-41aa-b744-e43b9fef5d59","Type":"ContainerDied","Data":"3422e9aceec1c4508d5494e939281f24be27498b415a0c4f9a87888ab86cc810"}
Jul 17 13:29:20.080350 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:20.079973    2565 scope.go:117] "RemoveContainer" containerID="af776563f25ac48ca4cbb3fa653f56010c2cd24c882dbe1cade4b620c511dd11"
Jul 17 13:29:20.080350 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:20.080294    2565 scope.go:117] "RemoveContainer" containerID="3422e9aceec1c4508d5494e939281f24be27498b415a0c4f9a87888ab86cc810"
Jul 17 13:29:20.080589 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:29:20.080564    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8_odh-ai-gateway-infra(5e34a350-9601-41aa-b744-e43b9fef5d59)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8" podUID="5e34a350-9601-41aa-b744-e43b9fef5d59"
Jul 17 13:29:20.081591 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:20.081575    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf"
Jul 17 13:29:20.081689 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:20.081619    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf" event={"ID":"ce9463ee-1100-445a-8744-0584bb5f138f","Type":"ContainerDied","Data":"3fdab60641471637b8e1ff020c7232b713c4f9952bda732e464cc297011b9824"}
Jul 17 13:29:20.093956 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:20.093938    2565 scope.go:117] "RemoveContainer" containerID="7b2002efc0019503587490a84effea6f4111320d773f0155eef8a6a864eab7d9"
Jul 17 13:29:20.098793 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:20.098769    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8"]
Jul 17 13:29:20.128594 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:20.128541    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf"]
Jul 17 13:29:20.142421 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:20.142370    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-gzncf"]
Jul 17 13:29:20.403945 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:20.403863    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce9463ee-1100-445a-8744-0584bb5f138f" path="/var/lib/kubelet/pods/ce9463ee-1100-445a-8744-0584bb5f138f/volumes"
Jul 17 13:29:21.239938 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:21.239914    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8"
Jul 17 13:29:21.254766 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:21.254737    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e34a350-9601-41aa-b744-e43b9fef5d59-maas-api-service-ca\") pod \"5e34a350-9601-41aa-b744-e43b9fef5d59\" (UID: \"5e34a350-9601-41aa-b744-e43b9fef5d59\") "
Jul 17 13:29:21.255274 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:21.255249    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e34a350-9601-41aa-b744-e43b9fef5d59-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e34a350-9601-41aa-b744-e43b9fef5d59" (UID: "5e34a350-9601-41aa-b744-e43b9fef5d59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:29:21.356225 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:21.356147    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e34a350-9601-41aa-b744-e43b9fef5d59-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:29:22.097628 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:22.097600    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8"
Jul 17 13:29:22.097803 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:22.097638    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8" event={"ID":"5e34a350-9601-41aa-b744-e43b9fef5d59","Type":"ContainerDied","Data":"cd79097679d177283b0913c308a67f83d0bc612f8507d6c167cdd6873d31cc29"}
Jul 17 13:29:22.097803 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:22.097701    2565 scope.go:117] "RemoveContainer" containerID="3422e9aceec1c4508d5494e939281f24be27498b415a0c4f9a87888ab86cc810"
Jul 17 13:29:22.120475 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:22.120452    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8"]
Jul 17 13:29:22.128954 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:22.128931    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-9nxx8"]
Jul 17 13:29:22.403549 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:22.403483    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e34a350-9601-41aa-b744-e43b9fef5d59" path="/var/lib/kubelet/pods/5e34a350-9601-41aa-b744-e43b9fef5d59/volumes"
Jul 17 13:29:40.423430 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.423393    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m"]
Jul 17 13:29:40.424000 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.423962    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cc09458-b10f-4744-ac4d-b01b4af16f2c" containerName="revoke-keys"
Jul 17 13:29:40.424000 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424000    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cc09458-b10f-4744-ac4d-b01b4af16f2c" containerName="revoke-keys"
Jul 17 13:29:40.424158 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424014    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce9463ee-1100-445a-8744-0584bb5f138f" containerName="revoke-keys"
Jul 17 13:29:40.424158 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424022    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce9463ee-1100-445a-8744-0584bb5f138f" containerName="revoke-keys"
Jul 17 13:29:40.424158 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424037    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="847bd07a-fd90-4aa4-8fc8-2ad3e2142046" containerName="revoke-keys"
Jul 17 13:29:40.424158 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424045    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="847bd07a-fd90-4aa4-8fc8-2ad3e2142046" containerName="revoke-keys"
Jul 17 13:29:40.424158 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424052    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="847bd07a-fd90-4aa4-8fc8-2ad3e2142046" containerName="revoke-keys"
Jul 17 13:29:40.424158 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424060    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="847bd07a-fd90-4aa4-8fc8-2ad3e2142046" containerName="revoke-keys"
Jul 17 13:29:40.424158 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424067    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e34a350-9601-41aa-b744-e43b9fef5d59" containerName="revoke-keys"
Jul 17 13:29:40.424158 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424075    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e34a350-9601-41aa-b744-e43b9fef5d59" containerName="revoke-keys"
Jul 17 13:29:40.424158 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424093    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce9463ee-1100-445a-8744-0584bb5f138f" containerName="revoke-keys"
Jul 17 13:29:40.424158 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424101    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce9463ee-1100-445a-8744-0584bb5f138f" containerName="revoke-keys"
Jul 17 13:29:40.424158 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424110    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce9463ee-1100-445a-8744-0584bb5f138f" containerName="revoke-keys"
Jul 17 13:29:40.424158 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424117    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce9463ee-1100-445a-8744-0584bb5f138f" containerName="revoke-keys"
Jul 17 13:29:40.424158 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424149    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cc09458-b10f-4744-ac4d-b01b4af16f2c" containerName="revoke-keys"
Jul 17 13:29:40.424158 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424158    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cc09458-b10f-4744-ac4d-b01b4af16f2c" containerName="revoke-keys"
Jul 17 13:29:40.424816 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424166    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e34a350-9601-41aa-b744-e43b9fef5d59" containerName="revoke-keys"
Jul 17 13:29:40.424816 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424175    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e34a350-9601-41aa-b744-e43b9fef5d59" containerName="revoke-keys"
Jul 17 13:29:40.424816 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424282    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce9463ee-1100-445a-8744-0584bb5f138f" containerName="revoke-keys"
Jul 17 13:29:40.424816 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424297    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce9463ee-1100-445a-8744-0584bb5f138f" containerName="revoke-keys"
Jul 17 13:29:40.424816 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424308    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="847bd07a-fd90-4aa4-8fc8-2ad3e2142046" containerName="revoke-keys"
Jul 17 13:29:40.424816 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424315    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e34a350-9601-41aa-b744-e43b9fef5d59" containerName="revoke-keys"
Jul 17 13:29:40.424816 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424325    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce9463ee-1100-445a-8744-0584bb5f138f" containerName="revoke-keys"
Jul 17 13:29:40.424816 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424335    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e34a350-9601-41aa-b744-e43b9fef5d59" containerName="revoke-keys"
Jul 17 13:29:40.424816 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424344    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e34a350-9601-41aa-b744-e43b9fef5d59" containerName="revoke-keys"
Jul 17 13:29:40.424816 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424352    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="847bd07a-fd90-4aa4-8fc8-2ad3e2142046" containerName="revoke-keys"
Jul 17 13:29:40.424816 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424359    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cc09458-b10f-4744-ac4d-b01b4af16f2c" containerName="revoke-keys"
Jul 17 13:29:40.424816 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.424369    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cc09458-b10f-4744-ac4d-b01b4af16f2c" containerName="revoke-keys"
Jul 17 13:29:40.431663 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.431632    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m"
Jul 17 13:29:40.434486 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.434302    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:29:40.435969 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.435938    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m"]
Jul 17 13:29:40.513783 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.513759    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5a5bec-a522-4c0f-9fcd-44e9c8f6508a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m\" (UID: \"db5a5bec-a522-4c0f-9fcd-44e9c8f6508a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m"
Jul 17 13:29:40.614759 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.614732    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5a5bec-a522-4c0f-9fcd-44e9c8f6508a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m\" (UID: \"db5a5bec-a522-4c0f-9fcd-44e9c8f6508a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m"
Jul 17 13:29:40.615350 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.615332    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5a5bec-a522-4c0f-9fcd-44e9c8f6508a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m\" (UID: \"db5a5bec-a522-4c0f-9fcd-44e9c8f6508a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m"
Jul 17 13:29:40.744426 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.744364    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m"
Jul 17 13:29:40.897051 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:40.897028    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m"]
Jul 17 13:29:40.898179 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:29:40.898148    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb5a5bec_a522_4c0f_9fcd_44e9c8f6508a.slice/crio-504143fa7c3f7b14b4938bf02c8cd0937fae6d9accd304e804bf22c765ec68a7 WatchSource:0}: Error finding container 504143fa7c3f7b14b4938bf02c8cd0937fae6d9accd304e804bf22c765ec68a7: Status 404 returned error can't find the container with id 504143fa7c3f7b14b4938bf02c8cd0937fae6d9accd304e804bf22c765ec68a7
Jul 17 13:29:41.222064 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:41.222022    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m" event={"ID":"db5a5bec-a522-4c0f-9fcd-44e9c8f6508a","Type":"ContainerStarted","Data":"504143fa7c3f7b14b4938bf02c8cd0937fae6d9accd304e804bf22c765ec68a7"}
Jul 17 13:29:42.232911 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:42.232875    2565 generic.go:356] "Generic (PLEG): container finished" podID="db5a5bec-a522-4c0f-9fcd-44e9c8f6508a" containerID="9d418f3efd3a8991dbb8809a49a153396ed03eae79b34712b73ec4f33b6a0966" exitCode=6
Jul 17 13:29:42.233330 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:42.232966    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m" event={"ID":"db5a5bec-a522-4c0f-9fcd-44e9c8f6508a","Type":"ContainerDied","Data":"9d418f3efd3a8991dbb8809a49a153396ed03eae79b34712b73ec4f33b6a0966"}
Jul 17 13:29:42.233330 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:42.233137    2565 scope.go:117] "RemoveContainer" containerID="9d418f3efd3a8991dbb8809a49a153396ed03eae79b34712b73ec4f33b6a0966"
Jul 17 13:29:43.240860 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:43.240829    2565 generic.go:356] "Generic (PLEG): container finished" podID="db5a5bec-a522-4c0f-9fcd-44e9c8f6508a" containerID="40cbb42b5015b1a7a4559a5b3ef602c78aee0ea25e17c137475b745b5c2f199f" exitCode=6
Jul 17 13:29:43.241223 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:43.240907    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m" event={"ID":"db5a5bec-a522-4c0f-9fcd-44e9c8f6508a","Type":"ContainerDied","Data":"40cbb42b5015b1a7a4559a5b3ef602c78aee0ea25e17c137475b745b5c2f199f"}
Jul 17 13:29:43.241223 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:43.240963    2565 scope.go:117] "RemoveContainer" containerID="9d418f3efd3a8991dbb8809a49a153396ed03eae79b34712b73ec4f33b6a0966"
Jul 17 13:29:43.241223 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:43.241151    2565 scope.go:117] "RemoveContainer" containerID="40cbb42b5015b1a7a4559a5b3ef602c78aee0ea25e17c137475b745b5c2f199f"
Jul 17 13:29:43.241377 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:29:43.241346    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m_odh-ai-gateway-infra(db5a5bec-a522-4c0f-9fcd-44e9c8f6508a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m" podUID="db5a5bec-a522-4c0f-9fcd-44e9c8f6508a"
Jul 17 13:29:44.248938 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:44.248909    2565 scope.go:117] "RemoveContainer" containerID="40cbb42b5015b1a7a4559a5b3ef602c78aee0ea25e17c137475b745b5c2f199f"
Jul 17 13:29:44.249311 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:29:44.249098    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m_odh-ai-gateway-infra(db5a5bec-a522-4c0f-9fcd-44e9c8f6508a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m" podUID="db5a5bec-a522-4c0f-9fcd-44e9c8f6508a"
Jul 17 13:29:48.209993 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:48.209918    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:29:49.097253 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:49.097220    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:29:49.148657 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:49.148621    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq"]
Jul 17 13:29:49.149242 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:49.149221    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cc09458-b10f-4744-ac4d-b01b4af16f2c" containerName="revoke-keys"
Jul 17 13:29:49.149242 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:49.149243    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cc09458-b10f-4744-ac4d-b01b4af16f2c" containerName="revoke-keys"
Jul 17 13:29:49.149417 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:49.149256    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e34a350-9601-41aa-b744-e43b9fef5d59" containerName="revoke-keys"
Jul 17 13:29:49.149417 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:49.149264    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e34a350-9601-41aa-b744-e43b9fef5d59" containerName="revoke-keys"
Jul 17 13:29:49.149417 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:49.149288    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="847bd07a-fd90-4aa4-8fc8-2ad3e2142046" containerName="revoke-keys"
Jul 17 13:29:49.149417 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:49.149296    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="847bd07a-fd90-4aa4-8fc8-2ad3e2142046" containerName="revoke-keys"
Jul 17 13:29:49.149625 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:49.149424    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="847bd07a-fd90-4aa4-8fc8-2ad3e2142046" containerName="revoke-keys"
Jul 17 13:29:49.149625 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:49.149440    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cc09458-b10f-4744-ac4d-b01b4af16f2c" containerName="revoke-keys"
Jul 17 13:29:49.154265 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:49.154246    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq"
Jul 17 13:29:49.160004 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:49.159853    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq"]
Jul 17 13:29:49.295525 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:49.295494    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/defef1da-2fdb-4be9-8a50-28a9df4db565-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq\" (UID: \"defef1da-2fdb-4be9-8a50-28a9df4db565\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq"
Jul 17 13:29:49.396900 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:49.396836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/defef1da-2fdb-4be9-8a50-28a9df4db565-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq\" (UID: \"defef1da-2fdb-4be9-8a50-28a9df4db565\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq"
Jul 17 13:29:49.397472 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:49.397453    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/defef1da-2fdb-4be9-8a50-28a9df4db565-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq\" (UID: \"defef1da-2fdb-4be9-8a50-28a9df4db565\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq"
Jul 17 13:29:49.466626 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:49.466589    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq"
Jul 17 13:29:49.604836 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:29:49.604808    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddefef1da_2fdb_4be9_8a50_28a9df4db565.slice/crio-716900ae50b655d732a5b7575ff990a78f3cd55df486a825fcb45f4fe4676476 WatchSource:0}: Error finding container 716900ae50b655d732a5b7575ff990a78f3cd55df486a825fcb45f4fe4676476: Status 404 returned error can't find the container with id 716900ae50b655d732a5b7575ff990a78f3cd55df486a825fcb45f4fe4676476
Jul 17 13:29:49.605715 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:49.605687    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq"]
Jul 17 13:29:50.286768 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:50.286679    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq" event={"ID":"defef1da-2fdb-4be9-8a50-28a9df4db565","Type":"ContainerStarted","Data":"7ceedb60446fa2c93e99efbfbd4280cd0499e5b657dab1493ddf5d4bd19a79af"}
Jul 17 13:29:50.286768 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:50.286738    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq" event={"ID":"defef1da-2fdb-4be9-8a50-28a9df4db565","Type":"ContainerStarted","Data":"716900ae50b655d732a5b7575ff990a78f3cd55df486a825fcb45f4fe4676476"}
Jul 17 13:29:50.308180 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:50.308130    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq" podStartSLOduration=1.3081112400000001 podStartE2EDuration="1.30811124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:29:50.306166484 +0000 UTC m=+4034.437551170" watchObservedRunningTime="2026-07-17 13:29:50.30811124 +0000 UTC m=+4034.439495903"
Jul 17 13:29:51.295090 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:51.295050    2565 generic.go:356] "Generic (PLEG): container finished" podID="defef1da-2fdb-4be9-8a50-28a9df4db565" containerID="7ceedb60446fa2c93e99efbfbd4280cd0499e5b657dab1493ddf5d4bd19a79af" exitCode=6
Jul 17 13:29:51.295277 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:51.295095    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq" event={"ID":"defef1da-2fdb-4be9-8a50-28a9df4db565","Type":"ContainerDied","Data":"7ceedb60446fa2c93e99efbfbd4280cd0499e5b657dab1493ddf5d4bd19a79af"}
Jul 17 13:29:51.295346 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:51.295292    2565 scope.go:117] "RemoveContainer" containerID="7ceedb60446fa2c93e99efbfbd4280cd0499e5b657dab1493ddf5d4bd19a79af"
Jul 17 13:29:51.674037 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:51.674010    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45"]
Jul 17 13:29:51.677351 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:51.677336    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45"
Jul 17 13:29:51.685875 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:51.685852    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45"]
Jul 17 13:29:51.822246 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:51.822214    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e195b35-a28c-46ea-a0d4-93c8027a6ee7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45\" (UID: \"8e195b35-a28c-46ea-a0d4-93c8027a6ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45"
Jul 17 13:29:51.922888 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:51.922827    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e195b35-a28c-46ea-a0d4-93c8027a6ee7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45\" (UID: \"8e195b35-a28c-46ea-a0d4-93c8027a6ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45"
Jul 17 13:29:51.923431 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:51.923411    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e195b35-a28c-46ea-a0d4-93c8027a6ee7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45\" (UID: \"8e195b35-a28c-46ea-a0d4-93c8027a6ee7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45"
Jul 17 13:29:51.988549 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:51.988525    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45"
Jul 17 13:29:52.128297 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:52.128269    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45"]
Jul 17 13:29:52.128911 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:29:52.128886    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e195b35_a28c_46ea_a0d4_93c8027a6ee7.slice/crio-2013ad83eee7de9c759120832d142eaca7329174790e5bf5474b43c4e9064798 WatchSource:0}: Error finding container 2013ad83eee7de9c759120832d142eaca7329174790e5bf5474b43c4e9064798: Status 404 returned error can't find the container with id 2013ad83eee7de9c759120832d142eaca7329174790e5bf5474b43c4e9064798
Jul 17 13:29:52.302745 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:52.302714    2565 generic.go:356] "Generic (PLEG): container finished" podID="defef1da-2fdb-4be9-8a50-28a9df4db565" containerID="4b2be5535a1d0b3acfdb165305bb9e58630c51d17ac10a895c42abb087199d2f" exitCode=6
Jul 17 13:29:52.302873 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:52.302795    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq" event={"ID":"defef1da-2fdb-4be9-8a50-28a9df4db565","Type":"ContainerDied","Data":"4b2be5535a1d0b3acfdb165305bb9e58630c51d17ac10a895c42abb087199d2f"}
Jul 17 13:29:52.302873 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:52.302864    2565 scope.go:117] "RemoveContainer" containerID="7ceedb60446fa2c93e99efbfbd4280cd0499e5b657dab1493ddf5d4bd19a79af"
Jul 17 13:29:52.303087 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:52.303068    2565 scope.go:117] "RemoveContainer" containerID="4b2be5535a1d0b3acfdb165305bb9e58630c51d17ac10a895c42abb087199d2f"
Jul 17 13:29:52.303345 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:29:52.303319    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq_odh-ai-gateway-infra(defef1da-2fdb-4be9-8a50-28a9df4db565)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq" podUID="defef1da-2fdb-4be9-8a50-28a9df4db565"
Jul 17 13:29:52.304226 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:52.304206    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45" event={"ID":"8e195b35-a28c-46ea-a0d4-93c8027a6ee7","Type":"ContainerStarted","Data":"2013ad83eee7de9c759120832d142eaca7329174790e5bf5474b43c4e9064798"}
Jul 17 13:29:53.027471 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:53.027439    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw"]
Jul 17 13:29:53.031186 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:53.031169    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw"
Jul 17 13:29:53.039877 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:53.037681    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw"]
Jul 17 13:29:53.134837 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:53.134763    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30a5cf47-9265-439b-a0bf-aa49ed3d8603-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw\" (UID: \"30a5cf47-9265-439b-a0bf-aa49ed3d8603\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw"
Jul 17 13:29:53.236031 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:53.235974    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30a5cf47-9265-439b-a0bf-aa49ed3d8603-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw\" (UID: \"30a5cf47-9265-439b-a0bf-aa49ed3d8603\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw"
Jul 17 13:29:53.236525 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:53.236509    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30a5cf47-9265-439b-a0bf-aa49ed3d8603-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw\" (UID: \"30a5cf47-9265-439b-a0bf-aa49ed3d8603\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw"
Jul 17 13:29:53.313287 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:53.313258    2565 scope.go:117] "RemoveContainer" containerID="4b2be5535a1d0b3acfdb165305bb9e58630c51d17ac10a895c42abb087199d2f"
Jul 17 13:29:53.313487 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:29:53.313458    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq_odh-ai-gateway-infra(defef1da-2fdb-4be9-8a50-28a9df4db565)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq" podUID="defef1da-2fdb-4be9-8a50-28a9df4db565"
Jul 17 13:29:53.314743 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:53.314719    2565 generic.go:356] "Generic (PLEG): container finished" podID="8e195b35-a28c-46ea-a0d4-93c8027a6ee7" containerID="5878f332bba39f7b016056c6012257d2595923da5209e8ac1b4721b13102223b" exitCode=6
Jul 17 13:29:53.314856 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:53.314788    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45" event={"ID":"8e195b35-a28c-46ea-a0d4-93c8027a6ee7","Type":"ContainerDied","Data":"5878f332bba39f7b016056c6012257d2595923da5209e8ac1b4721b13102223b"}
Jul 17 13:29:53.314933 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:53.314919    2565 scope.go:117] "RemoveContainer" containerID="5878f332bba39f7b016056c6012257d2595923da5209e8ac1b4721b13102223b"
Jul 17 13:29:53.342688 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:53.342670    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw"
Jul 17 13:29:53.478048 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:53.478024    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw"]
Jul 17 13:29:53.479174 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:29:53.479145    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a5cf47_9265_439b_a0bf_aa49ed3d8603.slice/crio-82d74267cb9bca75a4ad305cb0513938b64880af96ad538b02700088cb46dba4 WatchSource:0}: Error finding container 82d74267cb9bca75a4ad305cb0513938b64880af96ad538b02700088cb46dba4: Status 404 returned error can't find the container with id 82d74267cb9bca75a4ad305cb0513938b64880af96ad538b02700088cb46dba4
Jul 17 13:29:53.799597 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:53.799571    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:29:54.322949 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:54.322911    2565 generic.go:356] "Generic (PLEG): container finished" podID="8e195b35-a28c-46ea-a0d4-93c8027a6ee7" containerID="155dafba9bac5395b981302b478e632d5767d5bb414247e10cf7301da41f50ef" exitCode=6
Jul 17 13:29:54.323382 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:54.323010    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45" event={"ID":"8e195b35-a28c-46ea-a0d4-93c8027a6ee7","Type":"ContainerDied","Data":"155dafba9bac5395b981302b478e632d5767d5bb414247e10cf7301da41f50ef"}
Jul 17 13:29:54.323382 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:54.323094    2565 scope.go:117] "RemoveContainer" containerID="5878f332bba39f7b016056c6012257d2595923da5209e8ac1b4721b13102223b"
Jul 17 13:29:54.323382 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:54.323345    2565 scope.go:117] "RemoveContainer" containerID="155dafba9bac5395b981302b478e632d5767d5bb414247e10cf7301da41f50ef"
Jul 17 13:29:54.323606 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:29:54.323586    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45_odh-ai-gateway-infra(8e195b35-a28c-46ea-a0d4-93c8027a6ee7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45" podUID="8e195b35-a28c-46ea-a0d4-93c8027a6ee7"
Jul 17 13:29:54.324745 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:54.324725    2565 generic.go:356] "Generic (PLEG): container finished" podID="30a5cf47-9265-439b-a0bf-aa49ed3d8603" containerID="caf7bb3dffb4f2b915dbf69d51ac628e3611f31d3a22e6203c56410b2ec0d134" exitCode=6
Jul 17 13:29:54.324847 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:54.324833    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw" event={"ID":"30a5cf47-9265-439b-a0bf-aa49ed3d8603","Type":"ContainerDied","Data":"caf7bb3dffb4f2b915dbf69d51ac628e3611f31d3a22e6203c56410b2ec0d134"}
Jul 17 13:29:54.324886 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:54.324873    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw" event={"ID":"30a5cf47-9265-439b-a0bf-aa49ed3d8603","Type":"ContainerStarted","Data":"82d74267cb9bca75a4ad305cb0513938b64880af96ad538b02700088cb46dba4"}
Jul 17 13:29:54.324936 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:54.324924    2565 scope.go:117] "RemoveContainer" containerID="caf7bb3dffb4f2b915dbf69d51ac628e3611f31d3a22e6203c56410b2ec0d134"
Jul 17 13:29:55.336476 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:55.336452    2565 scope.go:117] "RemoveContainer" containerID="155dafba9bac5395b981302b478e632d5767d5bb414247e10cf7301da41f50ef"
Jul 17 13:29:55.336887 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:29:55.336667    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45_odh-ai-gateway-infra(8e195b35-a28c-46ea-a0d4-93c8027a6ee7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45" podUID="8e195b35-a28c-46ea-a0d4-93c8027a6ee7"
Jul 17 13:29:55.338158 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:55.338132    2565 generic.go:356] "Generic (PLEG): container finished" podID="30a5cf47-9265-439b-a0bf-aa49ed3d8603" containerID="8acf088b25e4a47a6df225ad1e5d6aed1a5a7d686e356f15ef96340f92f80207" exitCode=6
Jul 17 13:29:55.338291 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:55.338211    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw" event={"ID":"30a5cf47-9265-439b-a0bf-aa49ed3d8603","Type":"ContainerDied","Data":"8acf088b25e4a47a6df225ad1e5d6aed1a5a7d686e356f15ef96340f92f80207"}
Jul 17 13:29:55.338291 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:55.338263    2565 scope.go:117] "RemoveContainer" containerID="caf7bb3dffb4f2b915dbf69d51ac628e3611f31d3a22e6203c56410b2ec0d134"
Jul 17 13:29:55.338427 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:55.338411    2565 scope.go:117] "RemoveContainer" containerID="8acf088b25e4a47a6df225ad1e5d6aed1a5a7d686e356f15ef96340f92f80207"
Jul 17 13:29:55.338625 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:29:55.338602    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw_odh-ai-gateway-infra(30a5cf47-9265-439b-a0bf-aa49ed3d8603)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw" podUID="30a5cf47-9265-439b-a0bf-aa49ed3d8603"
Jul 17 13:29:55.396862 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:55.396845    2565 scope.go:117] "RemoveContainer" containerID="40cbb42b5015b1a7a4559a5b3ef602c78aee0ea25e17c137475b745b5c2f199f"
Jul 17 13:29:56.345721 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:56.345695    2565 generic.go:356] "Generic (PLEG): container finished" podID="db5a5bec-a522-4c0f-9fcd-44e9c8f6508a" containerID="6e53a457907e2703cd7fe03e2aa40254321a29543d4b14ef07e5e38537390b29" exitCode=6
Jul 17 13:29:56.346191 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:56.345779    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m" event={"ID":"db5a5bec-a522-4c0f-9fcd-44e9c8f6508a","Type":"ContainerDied","Data":"6e53a457907e2703cd7fe03e2aa40254321a29543d4b14ef07e5e38537390b29"}
Jul 17 13:29:56.346191 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:56.345855    2565 scope.go:117] "RemoveContainer" containerID="40cbb42b5015b1a7a4559a5b3ef602c78aee0ea25e17c137475b745b5c2f199f"
Jul 17 13:29:56.346191 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:56.346153    2565 scope.go:117] "RemoveContainer" containerID="6e53a457907e2703cd7fe03e2aa40254321a29543d4b14ef07e5e38537390b29"
Jul 17 13:29:56.346403 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:29:56.346378    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m_odh-ai-gateway-infra(db5a5bec-a522-4c0f-9fcd-44e9c8f6508a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m" podUID="db5a5bec-a522-4c0f-9fcd-44e9c8f6508a"
Jul 17 13:29:56.347639 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:56.347622    2565 scope.go:117] "RemoveContainer" containerID="8acf088b25e4a47a6df225ad1e5d6aed1a5a7d686e356f15ef96340f92f80207"
Jul 17 13:29:56.347825 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:29:56.347804    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw_odh-ai-gateway-infra(30a5cf47-9265-439b-a0bf-aa49ed3d8603)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw" podUID="30a5cf47-9265-439b-a0bf-aa49ed3d8603"
Jul 17 13:29:57.374129 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:57.374102    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m"]
Jul 17 13:29:57.515295 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:57.515265    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m"
Jul 17 13:29:57.674216 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:57.674138    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5a5bec-a522-4c0f-9fcd-44e9c8f6508a-maas-api-service-ca\") pod \"db5a5bec-a522-4c0f-9fcd-44e9c8f6508a\" (UID: \"db5a5bec-a522-4c0f-9fcd-44e9c8f6508a\") "
Jul 17 13:29:57.674489 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:57.674467    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5a5bec-a522-4c0f-9fcd-44e9c8f6508a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "db5a5bec-a522-4c0f-9fcd-44e9c8f6508a" (UID: "db5a5bec-a522-4c0f-9fcd-44e9c8f6508a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:29:57.775675 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:57.775650    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db5a5bec-a522-4c0f-9fcd-44e9c8f6508a-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:29:58.362604 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:58.362583    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m"
Jul 17 13:29:58.362773 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:58.362579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m" event={"ID":"db5a5bec-a522-4c0f-9fcd-44e9c8f6508a","Type":"ContainerDied","Data":"504143fa7c3f7b14b4938bf02c8cd0937fae6d9accd304e804bf22c765ec68a7"}
Jul 17 13:29:58.362773 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:58.362720    2565 scope.go:117] "RemoveContainer" containerID="6e53a457907e2703cd7fe03e2aa40254321a29543d4b14ef07e5e38537390b29"
Jul 17 13:29:58.388029 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:58.388009    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m"]
Jul 17 13:29:58.393609 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:58.393589    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-fvt8m"]
Jul 17 13:29:58.403026 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:29:58.403007    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db5a5bec-a522-4c0f-9fcd-44e9c8f6508a" path="/var/lib/kubelet/pods/db5a5bec-a522-4c0f-9fcd-44e9c8f6508a/volumes"
Jul 17 13:30:00.145585 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.145550    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5c3765-29738250-7l6b7"]
Jul 17 13:30:00.146075 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.146061    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db5a5bec-a522-4c0f-9fcd-44e9c8f6508a" containerName="revoke-keys"
Jul 17 13:30:00.146133 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.146079    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="db5a5bec-a522-4c0f-9fcd-44e9c8f6508a" containerName="revoke-keys"
Jul 17 13:30:00.146133 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.146100    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db5a5bec-a522-4c0f-9fcd-44e9c8f6508a" containerName="revoke-keys"
Jul 17 13:30:00.146133 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.146105    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="db5a5bec-a522-4c0f-9fcd-44e9c8f6508a" containerName="revoke-keys"
Jul 17 13:30:00.146228 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.146174    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="db5a5bec-a522-4c0f-9fcd-44e9c8f6508a" containerName="revoke-keys"
Jul 17 13:30:00.146228 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.146186    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="db5a5bec-a522-4c0f-9fcd-44e9c8f6508a" containerName="revoke-keys"
Jul 17 13:30:00.146228 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.146193    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="db5a5bec-a522-4c0f-9fcd-44e9c8f6508a" containerName="revoke-keys"
Jul 17 13:30:00.150695 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.150677    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e92eab-29738250-chrr4"]
Jul 17 13:30:00.150813 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.150794    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5c3765-29738250-7l6b7"
Jul 17 13:30:00.151090 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.151078    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db5a5bec-a522-4c0f-9fcd-44e9c8f6508a" containerName="revoke-keys"
Jul 17 13:30:00.151136 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.151092    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="db5a5bec-a522-4c0f-9fcd-44e9c8f6508a" containerName="revoke-keys"
Jul 17 13:30:00.154111 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.154091    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738250-t7586"]
Jul 17 13:30:00.154222 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.154207    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e92eab-29738250-chrr4"
Jul 17 13:30:00.158615 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.158506    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5c3765-29738250-7l6b7"]
Jul 17 13:30:00.158781 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.158600    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738250-t7586"
Jul 17 13:30:00.161778 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.161758    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e92eab-29738250-chrr4"]
Jul 17 13:30:00.165301 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.165278    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738250-t7586"]
Jul 17 13:30:00.165727 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.165707    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5c3765-29738250-7l6b7"
Jul 17 13:30:00.176691 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.176670    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e92eab-29738250-chrr4"
Jul 17 13:30:00.220379 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.220308    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738250-t7586"
Jul 17 13:30:00.343360 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.343329    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5c3765-29738250-7l6b7"]
Jul 17 13:30:00.343762 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:30:00.343706    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f0de93_703d_48eb_b7cb_aaeb6f9cc570.slice/crio-9d1e52e3442aeea93761c77953c01e0c6404eda773a756e17fdde06c4f8b6aee WatchSource:0}: Error finding container 9d1e52e3442aeea93761c77953c01e0c6404eda773a756e17fdde06c4f8b6aee: Status 404 returned error can't find the container with id 9d1e52e3442aeea93761c77953c01e0c6404eda773a756e17fdde06c4f8b6aee
Jul 17 13:30:00.375953 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.375922    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e92eab-29738250-chrr4"]
Jul 17 13:30:00.383026 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:30:00.382370    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5e0ac8_e141_4e9f_a2c1_a75e874f1deb.slice/crio-b8635dac5c0c7e05263ce3ad1f2d17cc71d0cb5c2e8886d536a4f7396fef2f99 WatchSource:0}: Error finding container b8635dac5c0c7e05263ce3ad1f2d17cc71d0cb5c2e8886d536a4f7396fef2f99: Status 404 returned error can't find the container with id b8635dac5c0c7e05263ce3ad1f2d17cc71d0cb5c2e8886d536a4f7396fef2f99
Jul 17 13:30:00.388485 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.388461    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5c3765-29738250-7l6b7" event={"ID":"80f0de93-703d-48eb-b7cb-aaeb6f9cc570","Type":"ContainerStarted","Data":"9d1e52e3442aeea93761c77953c01e0c6404eda773a756e17fdde06c4f8b6aee"}
Jul 17 13:30:00.409636 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:00.409605    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738250-t7586"]
Jul 17 13:30:00.412554 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:30:00.412496    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0491467e_4f8e_4913_991c_ec0ced8cb91c.slice/crio-9bd33f68430467c829e2ad358b4feb8750a00464f214f24506df932bcdf94a17 WatchSource:0}: Error finding container 9bd33f68430467c829e2ad358b4feb8750a00464f214f24506df932bcdf94a17: Status 404 returned error can't find the container with id 9bd33f68430467c829e2ad358b4feb8750a00464f214f24506df932bcdf94a17
Jul 17 13:30:01.396563 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:01.396530    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e92eab-29738250-chrr4" event={"ID":"cb5e0ac8-e141-4e9f-a2c1-a75e874f1deb","Type":"ContainerStarted","Data":"644968d6adce3dcb59626d3355529f1e2a682175b4dec58012fc7d255d8fcce3"}
Jul 17 13:30:01.396954 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:01.396579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e92eab-29738250-chrr4" event={"ID":"cb5e0ac8-e141-4e9f-a2c1-a75e874f1deb","Type":"ContainerStarted","Data":"b8635dac5c0c7e05263ce3ad1f2d17cc71d0cb5c2e8886d536a4f7396fef2f99"}
Jul 17 13:30:01.398324 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:01.398304    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738250-t7586" event={"ID":"0491467e-4f8e-4913-991c-ec0ced8cb91c","Type":"ContainerStarted","Data":"b2b5a330b9f09ed6addc3f2557e134fed0c7422de25fe7b9bfebebd2e4676114"}
Jul 17 13:30:01.398450 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:01.398345    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738250-t7586" event={"ID":"0491467e-4f8e-4913-991c-ec0ced8cb91c","Type":"ContainerStarted","Data":"9bd33f68430467c829e2ad358b4feb8750a00464f214f24506df932bcdf94a17"}
Jul 17 13:30:01.400074 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:01.400049    2565 generic.go:356] "Generic (PLEG): container finished" podID="80f0de93-703d-48eb-b7cb-aaeb6f9cc570" containerID="f847ca70770cc0544df15a835d4f283756cb2c2080c0551e1c65067880be33c0" exitCode=0
Jul 17 13:30:01.400144 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:01.400098    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5c3765-29738250-7l6b7" event={"ID":"80f0de93-703d-48eb-b7cb-aaeb6f9cc570","Type":"ContainerDied","Data":"f847ca70770cc0544df15a835d4f283756cb2c2080c0551e1c65067880be33c0"}
Jul 17 13:30:01.415420 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:01.415370    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e92eab-29738250-chrr4" podStartSLOduration=1.415353228 podStartE2EDuration="1.415353228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:30:01.411237843 +0000 UTC m=+4045.542622505" watchObservedRunningTime="2026-07-17 13:30:01.415353228 +0000 UTC m=+4045.546737884"
Jul 17 13:30:02.407720 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:02.407690    2565 generic.go:356] "Generic (PLEG): container finished" podID="cb5e0ac8-e141-4e9f-a2c1-a75e874f1deb" containerID="644968d6adce3dcb59626d3355529f1e2a682175b4dec58012fc7d255d8fcce3" exitCode=0
Jul 17 13:30:02.408216 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:02.407789    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e92eab-29738250-chrr4" event={"ID":"cb5e0ac8-e141-4e9f-a2c1-a75e874f1deb","Type":"ContainerDied","Data":"644968d6adce3dcb59626d3355529f1e2a682175b4dec58012fc7d255d8fcce3"}
Jul 17 13:30:02.409571 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:02.409544    2565 generic.go:356] "Generic (PLEG): container finished" podID="0491467e-4f8e-4913-991c-ec0ced8cb91c" containerID="b2b5a330b9f09ed6addc3f2557e134fed0c7422de25fe7b9bfebebd2e4676114" exitCode=0
Jul 17 13:30:02.409696 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:02.409609    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738250-t7586" event={"ID":"0491467e-4f8e-4913-991c-ec0ced8cb91c","Type":"ContainerDied","Data":"b2b5a330b9f09ed6addc3f2557e134fed0c7422de25fe7b9bfebebd2e4676114"}
Jul 17 13:30:02.424542 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:02.424496    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738250-t7586" podStartSLOduration=2.424481035 podStartE2EDuration="2.424481035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:30:01.440221688 +0000 UTC m=+4045.571606350" watchObservedRunningTime="2026-07-17 13:30:02.424481035 +0000 UTC m=+4046.555865696"
Jul 17 13:30:02.558732 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:02.558709    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5c3765-29738250-7l6b7"
Jul 17 13:30:03.417237 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:03.417209    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5c3765-29738250-7l6b7"
Jul 17 13:30:03.417682 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:03.417259    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5c3765-29738250-7l6b7" event={"ID":"80f0de93-703d-48eb-b7cb-aaeb6f9cc570","Type":"ContainerDied","Data":"9d1e52e3442aeea93761c77953c01e0c6404eda773a756e17fdde06c4f8b6aee"}
Jul 17 13:30:03.417682 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:03.417305    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d1e52e3442aeea93761c77953c01e0c6404eda773a756e17fdde06c4f8b6aee"
Jul 17 13:30:03.556803 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:03.556775    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e92eab-29738250-chrr4"
Jul 17 13:30:03.590774 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:03.590748    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738250-t7586"
Jul 17 13:30:04.425468 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:04.425447    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e92eab-29738250-chrr4"
Jul 17 13:30:04.425845 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:04.425487    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e92eab-29738250-chrr4" event={"ID":"cb5e0ac8-e141-4e9f-a2c1-a75e874f1deb","Type":"ContainerDied","Data":"b8635dac5c0c7e05263ce3ad1f2d17cc71d0cb5c2e8886d536a4f7396fef2f99"}
Jul 17 13:30:04.425845 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:04.425537    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8635dac5c0c7e05263ce3ad1f2d17cc71d0cb5c2e8886d536a4f7396fef2f99"
Jul 17 13:30:04.426880 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:04.426865    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738250-t7586"
Jul 17 13:30:04.427011 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:04.426928    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738250-t7586" event={"ID":"0491467e-4f8e-4913-991c-ec0ced8cb91c","Type":"ContainerDied","Data":"9bd33f68430467c829e2ad358b4feb8750a00464f214f24506df932bcdf94a17"}
Jul 17 13:30:04.427011 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:04.427000    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bd33f68430467c829e2ad358b4feb8750a00464f214f24506df932bcdf94a17"
Jul 17 13:30:07.397137 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:07.397111    2565 scope.go:117] "RemoveContainer" containerID="155dafba9bac5395b981302b478e632d5767d5bb414247e10cf7301da41f50ef"
Jul 17 13:30:08.397039 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:08.397012    2565 scope.go:117] "RemoveContainer" containerID="4b2be5535a1d0b3acfdb165305bb9e58630c51d17ac10a895c42abb087199d2f"
Jul 17 13:30:08.453661 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:08.453636    2565 generic.go:356] "Generic (PLEG): container finished" podID="8e195b35-a28c-46ea-a0d4-93c8027a6ee7" containerID="3db5e4df66ffd293070db672c477734e6a310ce41caa51ea4038fcc8780cc984" exitCode=6
Jul 17 13:30:08.453941 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:08.453706    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45" event={"ID":"8e195b35-a28c-46ea-a0d4-93c8027a6ee7","Type":"ContainerDied","Data":"3db5e4df66ffd293070db672c477734e6a310ce41caa51ea4038fcc8780cc984"}
Jul 17 13:30:08.453941 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:08.453757    2565 scope.go:117] "RemoveContainer" containerID="155dafba9bac5395b981302b478e632d5767d5bb414247e10cf7301da41f50ef"
Jul 17 13:30:08.454118 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:08.454102    2565 scope.go:117] "RemoveContainer" containerID="3db5e4df66ffd293070db672c477734e6a310ce41caa51ea4038fcc8780cc984"
Jul 17 13:30:08.454332 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:30:08.454310    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45_odh-ai-gateway-infra(8e195b35-a28c-46ea-a0d4-93c8027a6ee7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45" podUID="8e195b35-a28c-46ea-a0d4-93c8027a6ee7"
Jul 17 13:30:09.461666 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:09.461637    2565 generic.go:356] "Generic (PLEG): container finished" podID="defef1da-2fdb-4be9-8a50-28a9df4db565" containerID="9477d1b20d57a9fb62d2d0e77cc791d1b13c25efac808a345646b85f2fbd3940" exitCode=6
Jul 17 13:30:09.462089 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:09.461705    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq" event={"ID":"defef1da-2fdb-4be9-8a50-28a9df4db565","Type":"ContainerDied","Data":"9477d1b20d57a9fb62d2d0e77cc791d1b13c25efac808a345646b85f2fbd3940"}
Jul 17 13:30:09.462089 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:09.461769    2565 scope.go:117] "RemoveContainer" containerID="4b2be5535a1d0b3acfdb165305bb9e58630c51d17ac10a895c42abb087199d2f"
Jul 17 13:30:09.462089 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:09.462068    2565 scope.go:117] "RemoveContainer" containerID="9477d1b20d57a9fb62d2d0e77cc791d1b13c25efac808a345646b85f2fbd3940"
Jul 17 13:30:09.462309 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:30:09.462290    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq_odh-ai-gateway-infra(defef1da-2fdb-4be9-8a50-28a9df4db565)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq" podUID="defef1da-2fdb-4be9-8a50-28a9df4db565"
Jul 17 13:30:09.485148 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:09.485124    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45"]
Jul 17 13:30:09.630180 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:09.630157    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45"
Jul 17 13:30:09.681752 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:09.681728    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e195b35-a28c-46ea-a0d4-93c8027a6ee7-maas-api-service-ca\") pod \"8e195b35-a28c-46ea-a0d4-93c8027a6ee7\" (UID: \"8e195b35-a28c-46ea-a0d4-93c8027a6ee7\") "
Jul 17 13:30:09.682053 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:09.682034    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e195b35-a28c-46ea-a0d4-93c8027a6ee7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8e195b35-a28c-46ea-a0d4-93c8027a6ee7" (UID: "8e195b35-a28c-46ea-a0d4-93c8027a6ee7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:30:09.782335 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:09.782283    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e195b35-a28c-46ea-a0d4-93c8027a6ee7-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:30:10.396637 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:10.396609    2565 scope.go:117] "RemoveContainer" containerID="8acf088b25e4a47a6df225ad1e5d6aed1a5a7d686e356f15ef96340f92f80207"
Jul 17 13:30:10.472275 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:10.472252    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45"
Jul 17 13:30:10.472598 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:10.472299    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45" event={"ID":"8e195b35-a28c-46ea-a0d4-93c8027a6ee7","Type":"ContainerDied","Data":"2013ad83eee7de9c759120832d142eaca7329174790e5bf5474b43c4e9064798"}
Jul 17 13:30:10.472598 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:10.472362    2565 scope.go:117] "RemoveContainer" containerID="3db5e4df66ffd293070db672c477734e6a310ce41caa51ea4038fcc8780cc984"
Jul 17 13:30:10.490315 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:10.490294    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45"]
Jul 17 13:30:10.495518 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:10.495499    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq"]
Jul 17 13:30:10.500056 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:10.500038    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-hwx45"]
Jul 17 13:30:10.643498 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:10.643472    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq"
Jul 17 13:30:10.688300 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:10.688234    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/defef1da-2fdb-4be9-8a50-28a9df4db565-maas-api-service-ca\") pod \"defef1da-2fdb-4be9-8a50-28a9df4db565\" (UID: \"defef1da-2fdb-4be9-8a50-28a9df4db565\") "
Jul 17 13:30:10.688597 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:10.688577    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/defef1da-2fdb-4be9-8a50-28a9df4db565-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "defef1da-2fdb-4be9-8a50-28a9df4db565" (UID: "defef1da-2fdb-4be9-8a50-28a9df4db565"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:30:10.789640 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:10.789618    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/defef1da-2fdb-4be9-8a50-28a9df4db565-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:30:11.480621 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:11.480589    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq"
Jul 17 13:30:11.480621 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:11.480612    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq" event={"ID":"defef1da-2fdb-4be9-8a50-28a9df4db565","Type":"ContainerDied","Data":"716900ae50b655d732a5b7575ff990a78f3cd55df486a825fcb45f4fe4676476"}
Jul 17 13:30:11.481105 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:11.480673    2565 scope.go:117] "RemoveContainer" containerID="9477d1b20d57a9fb62d2d0e77cc791d1b13c25efac808a345646b85f2fbd3940"
Jul 17 13:30:11.483798 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:11.483763    2565 generic.go:356] "Generic (PLEG): container finished" podID="30a5cf47-9265-439b-a0bf-aa49ed3d8603" containerID="ac685194c79755de33c359d2eb14840e462cb9301767f1b4c8904e8881e1acd8" exitCode=6
Jul 17 13:30:11.483896 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:11.483799    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw" event={"ID":"30a5cf47-9265-439b-a0bf-aa49ed3d8603","Type":"ContainerDied","Data":"ac685194c79755de33c359d2eb14840e462cb9301767f1b4c8904e8881e1acd8"}
Jul 17 13:30:11.484153 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:11.484137    2565 scope.go:117] "RemoveContainer" containerID="ac685194c79755de33c359d2eb14840e462cb9301767f1b4c8904e8881e1acd8"
Jul 17 13:30:11.484379 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:30:11.484358    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw_odh-ai-gateway-infra(30a5cf47-9265-439b-a0bf-aa49ed3d8603)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw" podUID="30a5cf47-9265-439b-a0bf-aa49ed3d8603"
Jul 17 13:30:11.490566 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:11.490552    2565 scope.go:117] "RemoveContainer" containerID="8acf088b25e4a47a6df225ad1e5d6aed1a5a7d686e356f15ef96340f92f80207"
Jul 17 13:30:11.516074 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:11.516054    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq"]
Jul 17 13:30:11.522642 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:11.522623    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-hxdhq"]
Jul 17 13:30:12.402830 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:12.402791    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e195b35-a28c-46ea-a0d4-93c8027a6ee7" path="/var/lib/kubelet/pods/8e195b35-a28c-46ea-a0d4-93c8027a6ee7/volumes"
Jul 17 13:30:12.403153 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:12.403137    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="defef1da-2fdb-4be9-8a50-28a9df4db565" path="/var/lib/kubelet/pods/defef1da-2fdb-4be9-8a50-28a9df4db565/volumes"
Jul 17 13:30:12.512822 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:12.512793    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw"]
Jul 17 13:30:12.654465 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:12.654411    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw"
Jul 17 13:30:12.704682 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:12.704662    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30a5cf47-9265-439b-a0bf-aa49ed3d8603-maas-api-service-ca\") pod \"30a5cf47-9265-439b-a0bf-aa49ed3d8603\" (UID: \"30a5cf47-9265-439b-a0bf-aa49ed3d8603\") "
Jul 17 13:30:12.705001 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:12.704963    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a5cf47-9265-439b-a0bf-aa49ed3d8603-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "30a5cf47-9265-439b-a0bf-aa49ed3d8603" (UID: "30a5cf47-9265-439b-a0bf-aa49ed3d8603"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:30:12.805247 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:12.805212    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30a5cf47-9265-439b-a0bf-aa49ed3d8603-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:30:13.500111 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:13.500088    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw"
Jul 17 13:30:13.500281 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:13.500080    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw" event={"ID":"30a5cf47-9265-439b-a0bf-aa49ed3d8603","Type":"ContainerDied","Data":"82d74267cb9bca75a4ad305cb0513938b64880af96ad538b02700088cb46dba4"}
Jul 17 13:30:13.500281 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:13.500232    2565 scope.go:117] "RemoveContainer" containerID="ac685194c79755de33c359d2eb14840e462cb9301767f1b4c8904e8881e1acd8"
Jul 17 13:30:13.522819 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:13.522796    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw"]
Jul 17 13:30:13.526873 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:13.526850    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-m5ddw"]
Jul 17 13:30:14.403323 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:14.403288    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30a5cf47-9265-439b-a0bf-aa49ed3d8603" path="/var/lib/kubelet/pods/30a5cf47-9265-439b-a0bf-aa49ed3d8603/volumes"
Jul 17 13:30:15.401730 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:15.401702    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:30:30.482400 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.482368    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb"]
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.482843    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb5e0ac8-e141-4e9f-a2c1-a75e874f1deb" containerName="cleanup"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.482854    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb5e0ac8-e141-4e9f-a2c1-a75e874f1deb" containerName="cleanup"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.482864    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a5cf47-9265-439b-a0bf-aa49ed3d8603" containerName="revoke-keys"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.482869    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a5cf47-9265-439b-a0bf-aa49ed3d8603" containerName="revoke-keys"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.482876    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e195b35-a28c-46ea-a0d4-93c8027a6ee7" containerName="revoke-keys"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.482886    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e195b35-a28c-46ea-a0d4-93c8027a6ee7" containerName="revoke-keys"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.482897    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="defef1da-2fdb-4be9-8a50-28a9df4db565" containerName="revoke-keys"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.482902    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="defef1da-2fdb-4be9-8a50-28a9df4db565" containerName="revoke-keys"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.482911    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a5cf47-9265-439b-a0bf-aa49ed3d8603" containerName="revoke-keys"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.482916    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a5cf47-9265-439b-a0bf-aa49ed3d8603" containerName="revoke-keys"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.482923    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e195b35-a28c-46ea-a0d4-93c8027a6ee7" containerName="revoke-keys"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.482928    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e195b35-a28c-46ea-a0d4-93c8027a6ee7" containerName="revoke-keys"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.482934    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0491467e-4f8e-4913-991c-ec0ced8cb91c" containerName="cleanup"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.482940    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0491467e-4f8e-4913-991c-ec0ced8cb91c" containerName="cleanup"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.482950    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80f0de93-703d-48eb-b7cb-aaeb6f9cc570" containerName="cleanup"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.482955    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="80f0de93-703d-48eb-b7cb-aaeb6f9cc570" containerName="cleanup"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.482992    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="defef1da-2fdb-4be9-8a50-28a9df4db565" containerName="revoke-keys"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.482998    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="defef1da-2fdb-4be9-8a50-28a9df4db565" containerName="revoke-keys"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.483069    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="defef1da-2fdb-4be9-8a50-28a9df4db565" containerName="revoke-keys"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.483078    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="defef1da-2fdb-4be9-8a50-28a9df4db565" containerName="revoke-keys"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.483084    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="30a5cf47-9265-439b-a0bf-aa49ed3d8603" containerName="revoke-keys"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.483090    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e195b35-a28c-46ea-a0d4-93c8027a6ee7" containerName="revoke-keys"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.483094    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb5e0ac8-e141-4e9f-a2c1-a75e874f1deb" containerName="cleanup"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.483100    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="80f0de93-703d-48eb-b7cb-aaeb6f9cc570" containerName="cleanup"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.483106    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e195b35-a28c-46ea-a0d4-93c8027a6ee7" containerName="revoke-keys"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.483111    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="30a5cf47-9265-439b-a0bf-aa49ed3d8603" containerName="revoke-keys"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.483117    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="30a5cf47-9265-439b-a0bf-aa49ed3d8603" containerName="revoke-keys"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.483122    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e195b35-a28c-46ea-a0d4-93c8027a6ee7" containerName="revoke-keys"
Jul 17 13:30:30.485102 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.483128    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0491467e-4f8e-4913-991c-ec0ced8cb91c" containerName="cleanup"
Jul 17 13:30:30.486078 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.486063    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb"
Jul 17 13:30:30.488869 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.488847    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:30:30.496131 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.496109    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb"]
Jul 17 13:30:30.549999 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.549956    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d7c11f-30ba-4b45-aa8f-da310dcbe695-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb\" (UID: \"13d7c11f-30ba-4b45-aa8f-da310dcbe695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb"
Jul 17 13:30:30.651272 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.651238    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d7c11f-30ba-4b45-aa8f-da310dcbe695-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb\" (UID: \"13d7c11f-30ba-4b45-aa8f-da310dcbe695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb"
Jul 17 13:30:30.651798 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.651780    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d7c11f-30ba-4b45-aa8f-da310dcbe695-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb\" (UID: \"13d7c11f-30ba-4b45-aa8f-da310dcbe695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb"
Jul 17 13:30:30.797383 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.797362    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb"
Jul 17 13:30:30.933734 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:30.933701    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb"]
Jul 17 13:30:30.935562 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:30:30.935510    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d7c11f_30ba_4b45_aa8f_da310dcbe695.slice/crio-404e865187cb2134ac7b315fdcb9c0fd001ac9ce4d4fffa27ee6523941216b49 WatchSource:0}: Error finding container 404e865187cb2134ac7b315fdcb9c0fd001ac9ce4d4fffa27ee6523941216b49: Status 404 returned error can't find the container with id 404e865187cb2134ac7b315fdcb9c0fd001ac9ce4d4fffa27ee6523941216b49
Jul 17 13:30:31.627231 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:31.627121    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb" event={"ID":"13d7c11f-30ba-4b45-aa8f-da310dcbe695","Type":"ContainerStarted","Data":"fa03ee23987dd33d126a58cc55aa4222de57bccbbce1624610bccada5cc99e25"}
Jul 17 13:30:31.627561 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:31.627268    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb" event={"ID":"13d7c11f-30ba-4b45-aa8f-da310dcbe695","Type":"ContainerStarted","Data":"404e865187cb2134ac7b315fdcb9c0fd001ac9ce4d4fffa27ee6523941216b49"}
Jul 17 13:30:31.627722 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:31.627703    2565 scope.go:117] "RemoveContainer" containerID="fa03ee23987dd33d126a58cc55aa4222de57bccbbce1624610bccada5cc99e25"
Jul 17 13:30:32.636345 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:32.636314    2565 generic.go:356] "Generic (PLEG): container finished" podID="13d7c11f-30ba-4b45-aa8f-da310dcbe695" containerID="fa03ee23987dd33d126a58cc55aa4222de57bccbbce1624610bccada5cc99e25" exitCode=6
Jul 17 13:30:32.636345 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:32.636340    2565 generic.go:356] "Generic (PLEG): container finished" podID="13d7c11f-30ba-4b45-aa8f-da310dcbe695" containerID="edca7a87f34b083a1e8b54b9ce6a9f9d3d9a5417d913517f258695d4500467e1" exitCode=6
Jul 17 13:30:32.636795 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:32.636404    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb" event={"ID":"13d7c11f-30ba-4b45-aa8f-da310dcbe695","Type":"ContainerDied","Data":"fa03ee23987dd33d126a58cc55aa4222de57bccbbce1624610bccada5cc99e25"}
Jul 17 13:30:32.636795 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:32.636454    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb" event={"ID":"13d7c11f-30ba-4b45-aa8f-da310dcbe695","Type":"ContainerDied","Data":"edca7a87f34b083a1e8b54b9ce6a9f9d3d9a5417d913517f258695d4500467e1"}
Jul 17 13:30:32.636795 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:32.636480    2565 scope.go:117] "RemoveContainer" containerID="fa03ee23987dd33d126a58cc55aa4222de57bccbbce1624610bccada5cc99e25"
Jul 17 13:30:32.636795 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:32.636551    2565 scope.go:117] "RemoveContainer" containerID="edca7a87f34b083a1e8b54b9ce6a9f9d3d9a5417d913517f258695d4500467e1"
Jul 17 13:30:32.637021 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:30:32.636792    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb_odh-ai-gateway-infra(13d7c11f-30ba-4b45-aa8f-da310dcbe695)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb" podUID="13d7c11f-30ba-4b45-aa8f-da310dcbe695"
Jul 17 13:30:32.648131 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:32.648115    2565 scope.go:117] "RemoveContainer" containerID="fa03ee23987dd33d126a58cc55aa4222de57bccbbce1624610bccada5cc99e25"
Jul 17 13:30:32.648374 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:30:32.648355    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa03ee23987dd33d126a58cc55aa4222de57bccbbce1624610bccada5cc99e25\": container with ID starting with fa03ee23987dd33d126a58cc55aa4222de57bccbbce1624610bccada5cc99e25 not found: ID does not exist" containerID="fa03ee23987dd33d126a58cc55aa4222de57bccbbce1624610bccada5cc99e25"
Jul 17 13:30:32.648429 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:32.648385    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa03ee23987dd33d126a58cc55aa4222de57bccbbce1624610bccada5cc99e25"} err="failed to get container status \"fa03ee23987dd33d126a58cc55aa4222de57bccbbce1624610bccada5cc99e25\": rpc error: code = NotFound desc = could not find container \"fa03ee23987dd33d126a58cc55aa4222de57bccbbce1624610bccada5cc99e25\": container with ID starting with fa03ee23987dd33d126a58cc55aa4222de57bccbbce1624610bccada5cc99e25 not found: ID does not exist"
Jul 17 13:30:33.644502 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:33.644469    2565 scope.go:117] "RemoveContainer" containerID="edca7a87f34b083a1e8b54b9ce6a9f9d3d9a5417d913517f258695d4500467e1"
Jul 17 13:30:33.644914 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:30:33.644631    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb_odh-ai-gateway-infra(13d7c11f-30ba-4b45-aa8f-da310dcbe695)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb" podUID="13d7c11f-30ba-4b45-aa8f-da310dcbe695"
Jul 17 13:30:44.396806 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:44.396775    2565 scope.go:117] "RemoveContainer" containerID="edca7a87f34b083a1e8b54b9ce6a9f9d3d9a5417d913517f258695d4500467e1"
Jul 17 13:30:44.722589 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:44.722514    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb" event={"ID":"13d7c11f-30ba-4b45-aa8f-da310dcbe695","Type":"ContainerStarted","Data":"2260515426acb59ff053787b05134ce030f651ffc6d9e20b3cb565c00c167de8"}
Jul 17 13:30:44.739657 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:44.739610    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb" podStartSLOduration=14.739596287 podStartE2EDuration="14.739596287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:30:44.737068735 +0000 UTC m=+4088.868453419" watchObservedRunningTime="2026-07-17 13:30:44.739596287 +0000 UTC m=+4088.870980947"
Jul 17 13:30:45.733915 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:45.733884    2565 generic.go:356] "Generic (PLEG): container finished" podID="13d7c11f-30ba-4b45-aa8f-da310dcbe695" containerID="2260515426acb59ff053787b05134ce030f651ffc6d9e20b3cb565c00c167de8" exitCode=6
Jul 17 13:30:45.734373 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:45.733952    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb" event={"ID":"13d7c11f-30ba-4b45-aa8f-da310dcbe695","Type":"ContainerDied","Data":"2260515426acb59ff053787b05134ce030f651ffc6d9e20b3cb565c00c167de8"}
Jul 17 13:30:45.734373 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:45.734030    2565 scope.go:117] "RemoveContainer" containerID="edca7a87f34b083a1e8b54b9ce6a9f9d3d9a5417d913517f258695d4500467e1"
Jul 17 13:30:45.734373 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:45.734330    2565 scope.go:117] "RemoveContainer" containerID="2260515426acb59ff053787b05134ce030f651ffc6d9e20b3cb565c00c167de8"
Jul 17 13:30:45.734605 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:30:45.734581    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb_odh-ai-gateway-infra(13d7c11f-30ba-4b45-aa8f-da310dcbe695)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb" podUID="13d7c11f-30ba-4b45-aa8f-da310dcbe695"
Jul 17 13:30:45.749278 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:45.749259    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb"]
Jul 17 13:30:46.881034 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:46.881013    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb"
Jul 17 13:30:46.986621 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:46.986575    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d7c11f-30ba-4b45-aa8f-da310dcbe695-maas-api-service-ca\") pod \"13d7c11f-30ba-4b45-aa8f-da310dcbe695\" (UID: \"13d7c11f-30ba-4b45-aa8f-da310dcbe695\") "
Jul 17 13:30:46.986934 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:46.986913    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d7c11f-30ba-4b45-aa8f-da310dcbe695-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "13d7c11f-30ba-4b45-aa8f-da310dcbe695" (UID: "13d7c11f-30ba-4b45-aa8f-da310dcbe695"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:30:47.087942 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:47.087917    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d7c11f-30ba-4b45-aa8f-da310dcbe695-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:30:47.749908 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:47.749872    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb"
Jul 17 13:30:47.750116 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:47.749906    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb" event={"ID":"13d7c11f-30ba-4b45-aa8f-da310dcbe695","Type":"ContainerDied","Data":"404e865187cb2134ac7b315fdcb9c0fd001ac9ce4d4fffa27ee6523941216b49"}
Jul 17 13:30:47.750116 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:47.749970    2565 scope.go:117] "RemoveContainer" containerID="2260515426acb59ff053787b05134ce030f651ffc6d9e20b3cb565c00c167de8"
Jul 17 13:30:47.771913 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:47.771887    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb"]
Jul 17 13:30:47.775553 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:47.775530    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-blfwb"]
Jul 17 13:30:48.402857 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:48.402823    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13d7c11f-30ba-4b45-aa8f-da310dcbe695" path="/var/lib/kubelet/pods/13d7c11f-30ba-4b45-aa8f-da310dcbe695/volumes"
Jul 17 13:30:49.196706 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.196680    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s"]
Jul 17 13:30:49.197169 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.197153    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="defef1da-2fdb-4be9-8a50-28a9df4db565" containerName="revoke-keys"
Jul 17 13:30:49.197169 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.197170    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="defef1da-2fdb-4be9-8a50-28a9df4db565" containerName="revoke-keys"
Jul 17 13:30:49.197283 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.197187    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d7c11f-30ba-4b45-aa8f-da310dcbe695" containerName="revoke-keys"
Jul 17 13:30:49.197283 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.197192    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d7c11f-30ba-4b45-aa8f-da310dcbe695" containerName="revoke-keys"
Jul 17 13:30:49.197283 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.197199    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a5cf47-9265-439b-a0bf-aa49ed3d8603" containerName="revoke-keys"
Jul 17 13:30:49.197283 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.197204    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a5cf47-9265-439b-a0bf-aa49ed3d8603" containerName="revoke-keys"
Jul 17 13:30:49.197283 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.197212    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e195b35-a28c-46ea-a0d4-93c8027a6ee7" containerName="revoke-keys"
Jul 17 13:30:49.197283 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.197217    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e195b35-a28c-46ea-a0d4-93c8027a6ee7" containerName="revoke-keys"
Jul 17 13:30:49.197283 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.197237    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d7c11f-30ba-4b45-aa8f-da310dcbe695" containerName="revoke-keys"
Jul 17 13:30:49.197283 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.197242    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d7c11f-30ba-4b45-aa8f-da310dcbe695" containerName="revoke-keys"
Jul 17 13:30:49.197554 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.197313    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="13d7c11f-30ba-4b45-aa8f-da310dcbe695" containerName="revoke-keys"
Jul 17 13:30:49.197554 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.197323    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="13d7c11f-30ba-4b45-aa8f-da310dcbe695" containerName="revoke-keys"
Jul 17 13:30:49.197554 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.197329    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="13d7c11f-30ba-4b45-aa8f-da310dcbe695" containerName="revoke-keys"
Jul 17 13:30:49.197554 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.197338    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="defef1da-2fdb-4be9-8a50-28a9df4db565" containerName="revoke-keys"
Jul 17 13:30:49.201894 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.201876    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s"
Jul 17 13:30:49.208326 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.208300    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:30:49.212255 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.212230    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s"]
Jul 17 13:30:49.310610 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.310586    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87971599-16a5-4af4-9e51-6e9803096b2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s\" (UID: \"87971599-16a5-4af4-9e51-6e9803096b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s"
Jul 17 13:30:49.411668 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.411641    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87971599-16a5-4af4-9e51-6e9803096b2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s\" (UID: \"87971599-16a5-4af4-9e51-6e9803096b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s"
Jul 17 13:30:49.412251 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.412234    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87971599-16a5-4af4-9e51-6e9803096b2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s\" (UID: \"87971599-16a5-4af4-9e51-6e9803096b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s"
Jul 17 13:30:49.523174 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.523104    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s"
Jul 17 13:30:49.867844 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:49.867816    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s"]
Jul 17 13:30:49.868842 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:30:49.868819    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87971599_16a5_4af4_9e51_6e9803096b2a.slice/crio-39476b44280cbc444f4b4f5b186a1ae38a2e50a6f26dae2f8be546e4fb787aa0 WatchSource:0}: Error finding container 39476b44280cbc444f4b4f5b186a1ae38a2e50a6f26dae2f8be546e4fb787aa0: Status 404 returned error can't find the container with id 39476b44280cbc444f4b4f5b186a1ae38a2e50a6f26dae2f8be546e4fb787aa0
Jul 17 13:30:50.770955 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:50.770921    2565 generic.go:356] "Generic (PLEG): container finished" podID="87971599-16a5-4af4-9e51-6e9803096b2a" containerID="85fbe3c0629f8142087fefde02230a6ed092fc63f6d02765d0fb6600504cb36a" exitCode=6
Jul 17 13:30:50.771292 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:50.770963    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s" event={"ID":"87971599-16a5-4af4-9e51-6e9803096b2a","Type":"ContainerDied","Data":"85fbe3c0629f8142087fefde02230a6ed092fc63f6d02765d0fb6600504cb36a"}
Jul 17 13:30:50.771292 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:50.771039    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s" event={"ID":"87971599-16a5-4af4-9e51-6e9803096b2a","Type":"ContainerStarted","Data":"39476b44280cbc444f4b4f5b186a1ae38a2e50a6f26dae2f8be546e4fb787aa0"}
Jul 17 13:30:50.771292 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:50.771238    2565 scope.go:117] "RemoveContainer" containerID="85fbe3c0629f8142087fefde02230a6ed092fc63f6d02765d0fb6600504cb36a"
Jul 17 13:30:51.779458 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:51.779389    2565 generic.go:356] "Generic (PLEG): container finished" podID="87971599-16a5-4af4-9e51-6e9803096b2a" containerID="02bb6c2480a00bf7eb245417b92ac06954affde09d83d6eae0e58f7f7ba04cc1" exitCode=6
Jul 17 13:30:51.779872 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:51.779467    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s" event={"ID":"87971599-16a5-4af4-9e51-6e9803096b2a","Type":"ContainerDied","Data":"02bb6c2480a00bf7eb245417b92ac06954affde09d83d6eae0e58f7f7ba04cc1"}
Jul 17 13:30:51.779872 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:51.779525    2565 scope.go:117] "RemoveContainer" containerID="85fbe3c0629f8142087fefde02230a6ed092fc63f6d02765d0fb6600504cb36a"
Jul 17 13:30:51.779872 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:51.779725    2565 scope.go:117] "RemoveContainer" containerID="02bb6c2480a00bf7eb245417b92ac06954affde09d83d6eae0e58f7f7ba04cc1"
Jul 17 13:30:51.780068 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:30:51.779942    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s_odh-ai-gateway-infra(87971599-16a5-4af4-9e51-6e9803096b2a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s" podUID="87971599-16a5-4af4-9e51-6e9803096b2a"
Jul 17 13:30:52.787536 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:52.787511    2565 scope.go:117] "RemoveContainer" containerID="02bb6c2480a00bf7eb245417b92ac06954affde09d83d6eae0e58f7f7ba04cc1"
Jul 17 13:30:52.787876 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:30:52.787675    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s_odh-ai-gateway-infra(87971599-16a5-4af4-9e51-6e9803096b2a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s" podUID="87971599-16a5-4af4-9e51-6e9803096b2a"
Jul 17 13:30:56.923786 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:56.923755    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:30:57.865538 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:30:57.865509    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:31:02.807489 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:02.807457    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:31:06.400930 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:06.400902    2565 scope.go:117] "RemoveContainer" containerID="02bb6c2480a00bf7eb245417b92ac06954affde09d83d6eae0e58f7f7ba04cc1"
Jul 17 13:31:06.876724 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:06.876690    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s" event={"ID":"87971599-16a5-4af4-9e51-6e9803096b2a","Type":"ContainerStarted","Data":"86fe05e23a336a3eda0b3b8dc2013d80e2878f799827aeeb5ef62f18e16329fe"}
Jul 17 13:31:06.894284 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:06.894237    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s" podStartSLOduration=17.894223167 podStartE2EDuration="17.894223167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:31:06.891597584 +0000 UTC m=+4111.022982246" watchObservedRunningTime="2026-07-17 13:31:06.894223167 +0000 UTC m=+4111.025607849"
Jul 17 13:31:07.885040 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:07.885001    2565 generic.go:356] "Generic (PLEG): container finished" podID="87971599-16a5-4af4-9e51-6e9803096b2a" containerID="86fe05e23a336a3eda0b3b8dc2013d80e2878f799827aeeb5ef62f18e16329fe" exitCode=6
Jul 17 13:31:07.885481 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:07.885019    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s" event={"ID":"87971599-16a5-4af4-9e51-6e9803096b2a","Type":"ContainerDied","Data":"86fe05e23a336a3eda0b3b8dc2013d80e2878f799827aeeb5ef62f18e16329fe"}
Jul 17 13:31:07.885481 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:07.885162    2565 scope.go:117] "RemoveContainer" containerID="02bb6c2480a00bf7eb245417b92ac06954affde09d83d6eae0e58f7f7ba04cc1"
Jul 17 13:31:07.885481 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:07.885464    2565 scope.go:117] "RemoveContainer" containerID="86fe05e23a336a3eda0b3b8dc2013d80e2878f799827aeeb5ef62f18e16329fe"
Jul 17 13:31:07.885735 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:31:07.885714    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s_odh-ai-gateway-infra(87971599-16a5-4af4-9e51-6e9803096b2a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s" podUID="87971599-16a5-4af4-9e51-6e9803096b2a"
Jul 17 13:31:07.902342 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:07.902316    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s"]
Jul 17 13:31:09.035301 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:09.035277    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s"
Jul 17 13:31:09.080644 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:09.080614    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87971599-16a5-4af4-9e51-6e9803096b2a-maas-api-service-ca\") pod \"87971599-16a5-4af4-9e51-6e9803096b2a\" (UID: \"87971599-16a5-4af4-9e51-6e9803096b2a\") "
Jul 17 13:31:09.080954 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:09.080929    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87971599-16a5-4af4-9e51-6e9803096b2a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "87971599-16a5-4af4-9e51-6e9803096b2a" (UID: "87971599-16a5-4af4-9e51-6e9803096b2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:31:09.181686 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:09.181599    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87971599-16a5-4af4-9e51-6e9803096b2a-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:31:09.900332 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:09.900286    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s"
Jul 17 13:31:09.900332 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:09.900312    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s" event={"ID":"87971599-16a5-4af4-9e51-6e9803096b2a","Type":"ContainerDied","Data":"39476b44280cbc444f4b4f5b186a1ae38a2e50a6f26dae2f8be546e4fb787aa0"}
Jul 17 13:31:09.900598 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:09.900398    2565 scope.go:117] "RemoveContainer" containerID="86fe05e23a336a3eda0b3b8dc2013d80e2878f799827aeeb5ef62f18e16329fe"
Jul 17 13:31:09.928404 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:09.928374    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s"]
Jul 17 13:31:09.936283 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:09.936257    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-wv26s"]
Jul 17 13:31:09.947672 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:09.947646    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:31:10.045656 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:10.045626    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:31:10.403973 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:10.403942    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87971599-16a5-4af4-9e51-6e9803096b2a" path="/var/lib/kubelet/pods/87971599-16a5-4af4-9e51-6e9803096b2a/volumes"
Jul 17 13:31:20.533948 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:20.533912    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr"]
Jul 17 13:31:20.534352 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:20.534339    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d7c11f-30ba-4b45-aa8f-da310dcbe695" containerName="revoke-keys"
Jul 17 13:31:20.534352 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:20.534353    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d7c11f-30ba-4b45-aa8f-da310dcbe695" containerName="revoke-keys"
Jul 17 13:31:20.534427 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:20.534363    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87971599-16a5-4af4-9e51-6e9803096b2a" containerName="revoke-keys"
Jul 17 13:31:20.534427 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:20.534368    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="87971599-16a5-4af4-9e51-6e9803096b2a" containerName="revoke-keys"
Jul 17 13:31:20.534427 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:20.534383    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87971599-16a5-4af4-9e51-6e9803096b2a" containerName="revoke-keys"
Jul 17 13:31:20.534427 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:20.534388    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="87971599-16a5-4af4-9e51-6e9803096b2a" containerName="revoke-keys"
Jul 17 13:31:20.534427 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:20.534395    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87971599-16a5-4af4-9e51-6e9803096b2a" containerName="revoke-keys"
Jul 17 13:31:20.534427 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:20.534400    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="87971599-16a5-4af4-9e51-6e9803096b2a" containerName="revoke-keys"
Jul 17 13:31:20.534611 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:20.534469    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="87971599-16a5-4af4-9e51-6e9803096b2a" containerName="revoke-keys"
Jul 17 13:31:20.534611 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:20.534478    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="87971599-16a5-4af4-9e51-6e9803096b2a" containerName="revoke-keys"
Jul 17 13:31:20.538908 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:20.538892    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr"
Jul 17 13:31:20.542197 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:20.542171    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:31:20.544749 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:20.544727    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr"]
Jul 17 13:31:20.578376 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:20.578354    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c74fbe-904f-493d-bc89-c0b43914f955-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr\" (UID: \"92c74fbe-904f-493d-bc89-c0b43914f955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr"
Jul 17 13:31:20.679615 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:20.679586    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c74fbe-904f-493d-bc89-c0b43914f955-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr\" (UID: \"92c74fbe-904f-493d-bc89-c0b43914f955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr"
Jul 17 13:31:20.680217 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:20.680199    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c74fbe-904f-493d-bc89-c0b43914f955-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr\" (UID: \"92c74fbe-904f-493d-bc89-c0b43914f955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr"
Jul 17 13:31:20.851231 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:20.851194    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr"
Jul 17 13:31:20.993231 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:20.993208    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr"]
Jul 17 13:31:21.980306 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:21.980218    2565 generic.go:356] "Generic (PLEG): container finished" podID="92c74fbe-904f-493d-bc89-c0b43914f955" containerID="fb745588c283164797896e908c5d259c48eb60d1d9167d14f7872dd5b2da65c6" exitCode=6
Jul 17 13:31:21.980683 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:21.980349    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr" event={"ID":"92c74fbe-904f-493d-bc89-c0b43914f955","Type":"ContainerDied","Data":"fb745588c283164797896e908c5d259c48eb60d1d9167d14f7872dd5b2da65c6"}
Jul 17 13:31:21.980683 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:21.980444    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr" event={"ID":"92c74fbe-904f-493d-bc89-c0b43914f955","Type":"ContainerStarted","Data":"2a273a8666922b31dd296e81e6d021f4a7d1d1c0467e13936d46ebf2ef646abc"}
Jul 17 13:31:21.980683 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:21.980454    2565 scope.go:117] "RemoveContainer" containerID="fb745588c283164797896e908c5d259c48eb60d1d9167d14f7872dd5b2da65c6"
Jul 17 13:31:22.988241 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:22.988164    2565 generic.go:356] "Generic (PLEG): container finished" podID="92c74fbe-904f-493d-bc89-c0b43914f955" containerID="2a08f569966edc051de0c43a5ee3ae1546d1b8f093b0206dfe43000c04701554" exitCode=6
Jul 17 13:31:22.988241 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:22.988234    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr" event={"ID":"92c74fbe-904f-493d-bc89-c0b43914f955","Type":"ContainerDied","Data":"2a08f569966edc051de0c43a5ee3ae1546d1b8f093b0206dfe43000c04701554"}
Jul 17 13:31:22.988622 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:22.988290    2565 scope.go:117] "RemoveContainer" containerID="fb745588c283164797896e908c5d259c48eb60d1d9167d14f7872dd5b2da65c6"
Jul 17 13:31:22.988622 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:22.988461    2565 scope.go:117] "RemoveContainer" containerID="2a08f569966edc051de0c43a5ee3ae1546d1b8f093b0206dfe43000c04701554"
Jul 17 13:31:22.988703 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:31:22.988663    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr_odh-ai-gateway-infra(92c74fbe-904f-493d-bc89-c0b43914f955)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr" podUID="92c74fbe-904f-493d-bc89-c0b43914f955"
Jul 17 13:31:23.995807 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:23.995780    2565 scope.go:117] "RemoveContainer" containerID="2a08f569966edc051de0c43a5ee3ae1546d1b8f093b0206dfe43000c04701554"
Jul 17 13:31:23.996187 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:31:23.995956    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr_odh-ai-gateway-infra(92c74fbe-904f-493d-bc89-c0b43914f955)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr" podUID="92c74fbe-904f-493d-bc89-c0b43914f955"
Jul 17 13:31:31.370822 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:31.370742    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:31:37.397426 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:37.397390    2565 scope.go:117] "RemoveContainer" containerID="2a08f569966edc051de0c43a5ee3ae1546d1b8f093b0206dfe43000c04701554"
Jul 17 13:31:38.093530 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:38.093501    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr" event={"ID":"92c74fbe-904f-493d-bc89-c0b43914f955","Type":"ContainerStarted","Data":"206382122af66fc09e97f1dd47d7ac1e888499c31f1aed90a3eceaf416104ab5"}
Jul 17 13:31:38.111511 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:38.111446    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr" podStartSLOduration=18.111428303 podStartE2EDuration="18.111428303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:31:38.111101821 +0000 UTC m=+4142.242486481" watchObservedRunningTime="2026-07-17 13:31:38.111428303 +0000 UTC m=+4142.242812964"
Jul 17 13:31:39.100897 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:39.100868    2565 generic.go:356] "Generic (PLEG): container finished" podID="92c74fbe-904f-493d-bc89-c0b43914f955" containerID="206382122af66fc09e97f1dd47d7ac1e888499c31f1aed90a3eceaf416104ab5" exitCode=6
Jul 17 13:31:39.101360 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:39.100939    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr" event={"ID":"92c74fbe-904f-493d-bc89-c0b43914f955","Type":"ContainerDied","Data":"206382122af66fc09e97f1dd47d7ac1e888499c31f1aed90a3eceaf416104ab5"}
Jul 17 13:31:39.101360 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:39.101019    2565 scope.go:117] "RemoveContainer" containerID="2a08f569966edc051de0c43a5ee3ae1546d1b8f093b0206dfe43000c04701554"
Jul 17 13:31:39.101360 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:39.101355    2565 scope.go:117] "RemoveContainer" containerID="206382122af66fc09e97f1dd47d7ac1e888499c31f1aed90a3eceaf416104ab5"
Jul 17 13:31:39.101597 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:31:39.101572    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr_odh-ai-gateway-infra(92c74fbe-904f-493d-bc89-c0b43914f955)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr" podUID="92c74fbe-904f-493d-bc89-c0b43914f955"
Jul 17 13:31:39.134265 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:39.129388    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr"]
Jul 17 13:31:40.260583 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:40.260561    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr"
Jul 17 13:31:40.336363 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:40.336328    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c74fbe-904f-493d-bc89-c0b43914f955-maas-api-service-ca\") pod \"92c74fbe-904f-493d-bc89-c0b43914f955\" (UID: \"92c74fbe-904f-493d-bc89-c0b43914f955\") "
Jul 17 13:31:40.336632 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:40.336608    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c74fbe-904f-493d-bc89-c0b43914f955-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "92c74fbe-904f-493d-bc89-c0b43914f955" (UID: "92c74fbe-904f-493d-bc89-c0b43914f955"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:31:40.437714 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:40.437689    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c74fbe-904f-493d-bc89-c0b43914f955-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:31:41.118770 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:41.118738    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr"
Jul 17 13:31:41.118998 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:41.118784    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr" event={"ID":"92c74fbe-904f-493d-bc89-c0b43914f955","Type":"ContainerDied","Data":"2a273a8666922b31dd296e81e6d021f4a7d1d1c0467e13936d46ebf2ef646abc"}
Jul 17 13:31:41.118998 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:41.118858    2565 scope.go:117] "RemoveContainer" containerID="206382122af66fc09e97f1dd47d7ac1e888499c31f1aed90a3eceaf416104ab5"
Jul 17 13:31:41.137461 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:41.137432    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr"]
Jul 17 13:31:41.144958 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:41.144935    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-76kbr"]
Jul 17 13:31:41.767205 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:41.767171    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl"]
Jul 17 13:31:41.767647 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:41.767591    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92c74fbe-904f-493d-bc89-c0b43914f955" containerName="revoke-keys"
Jul 17 13:31:41.767647 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:41.767602    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="92c74fbe-904f-493d-bc89-c0b43914f955" containerName="revoke-keys"
Jul 17 13:31:41.767647 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:41.767631    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92c74fbe-904f-493d-bc89-c0b43914f955" containerName="revoke-keys"
Jul 17 13:31:41.767647 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:41.767636    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="92c74fbe-904f-493d-bc89-c0b43914f955" containerName="revoke-keys"
Jul 17 13:31:41.767838 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:41.767696    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="92c74fbe-904f-493d-bc89-c0b43914f955" containerName="revoke-keys"
Jul 17 13:31:41.767838 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:41.767707    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="87971599-16a5-4af4-9e51-6e9803096b2a" containerName="revoke-keys"
Jul 17 13:31:41.767838 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:41.767713    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="92c74fbe-904f-493d-bc89-c0b43914f955" containerName="revoke-keys"
Jul 17 13:31:41.772129 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:41.772110    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl"
Jul 17 13:31:41.775677 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:41.775640    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:31:41.780345 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:41.780306    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl"]
Jul 17 13:31:41.850526 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:41.850488    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c878ed16-8d2f-4635-b764-2051883bb31d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl\" (UID: \"c878ed16-8d2f-4635-b764-2051883bb31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl"
Jul 17 13:31:41.951405 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:41.951373    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c878ed16-8d2f-4635-b764-2051883bb31d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl\" (UID: \"c878ed16-8d2f-4635-b764-2051883bb31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl"
Jul 17 13:31:41.952021 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:41.951970    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c878ed16-8d2f-4635-b764-2051883bb31d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl\" (UID: \"c878ed16-8d2f-4635-b764-2051883bb31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl"
Jul 17 13:31:42.085913 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:42.085887    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl"
Jul 17 13:31:42.228760 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:42.228726    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl"]
Jul 17 13:31:42.229191 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:31:42.229167    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc878ed16_8d2f_4635_b764_2051883bb31d.slice/crio-6d0b963d2f830e3f65f2092c4392adff0c39226cdb5fb24be237b160fc281794 WatchSource:0}: Error finding container 6d0b963d2f830e3f65f2092c4392adff0c39226cdb5fb24be237b160fc281794: Status 404 returned error can't find the container with id 6d0b963d2f830e3f65f2092c4392adff0c39226cdb5fb24be237b160fc281794
Jul 17 13:31:42.403327 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:42.403233    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92c74fbe-904f-493d-bc89-c0b43914f955" path="/var/lib/kubelet/pods/92c74fbe-904f-493d-bc89-c0b43914f955/volumes"
Jul 17 13:31:43.109111 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:43.109037    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r"]
Jul 17 13:31:43.109536 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:43.109521    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92c74fbe-904f-493d-bc89-c0b43914f955" containerName="revoke-keys"
Jul 17 13:31:43.109576 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:43.109540    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="92c74fbe-904f-493d-bc89-c0b43914f955" containerName="revoke-keys"
Jul 17 13:31:43.109654 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:43.109644    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="92c74fbe-904f-493d-bc89-c0b43914f955" containerName="revoke-keys"
Jul 17 13:31:43.112893 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:43.112872    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r"
Jul 17 13:31:43.120062 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:43.120037    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r"]
Jul 17 13:31:43.135658 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:43.135631    2565 generic.go:356] "Generic (PLEG): container finished" podID="c878ed16-8d2f-4635-b764-2051883bb31d" containerID="1fe75d201e09e6877d05fde2d9bf72758a1ed2534e9e283ad13ea9576e263edb" exitCode=6
Jul 17 13:31:43.135776 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:43.135709    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl" event={"ID":"c878ed16-8d2f-4635-b764-2051883bb31d","Type":"ContainerDied","Data":"1fe75d201e09e6877d05fde2d9bf72758a1ed2534e9e283ad13ea9576e263edb"}
Jul 17 13:31:43.135776 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:43.135751    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl" event={"ID":"c878ed16-8d2f-4635-b764-2051883bb31d","Type":"ContainerStarted","Data":"6d0b963d2f830e3f65f2092c4392adff0c39226cdb5fb24be237b160fc281794"}
Jul 17 13:31:43.135941 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:43.135928    2565 scope.go:117] "RemoveContainer" containerID="1fe75d201e09e6877d05fde2d9bf72758a1ed2534e9e283ad13ea9576e263edb"
Jul 17 13:31:43.261570 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:43.261542    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eae707e-2cb7-47ca-949f-6d38a592f8dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r\" (UID: \"5eae707e-2cb7-47ca-949f-6d38a592f8dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r"
Jul 17 13:31:43.363110 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:43.363030    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eae707e-2cb7-47ca-949f-6d38a592f8dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r\" (UID: \"5eae707e-2cb7-47ca-949f-6d38a592f8dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r"
Jul 17 13:31:43.363622 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:43.363604    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eae707e-2cb7-47ca-949f-6d38a592f8dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r\" (UID: \"5eae707e-2cb7-47ca-949f-6d38a592f8dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r"
Jul 17 13:31:43.426522 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:43.426501    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r"
Jul 17 13:31:43.568439 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:43.568390    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r"]
Jul 17 13:31:43.574392 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:31:43.574364    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eae707e_2cb7_47ca_949f_6d38a592f8dc.slice/crio-82a597b28dc2e98a761b949196da23e43eb364650373fa79cc35c3b0bffc0a61 WatchSource:0}: Error finding container 82a597b28dc2e98a761b949196da23e43eb364650373fa79cc35c3b0bffc0a61: Status 404 returned error can't find the container with id 82a597b28dc2e98a761b949196da23e43eb364650373fa79cc35c3b0bffc0a61
Jul 17 13:31:44.144685 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:44.144648    2565 generic.go:356] "Generic (PLEG): container finished" podID="c878ed16-8d2f-4635-b764-2051883bb31d" containerID="8049d20707fc32aaa7147bddb8462e19b99b4ede814b33ea65e6ec2b3e6323ea" exitCode=6
Jul 17 13:31:44.145148 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:44.144724    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl" event={"ID":"c878ed16-8d2f-4635-b764-2051883bb31d","Type":"ContainerDied","Data":"8049d20707fc32aaa7147bddb8462e19b99b4ede814b33ea65e6ec2b3e6323ea"}
Jul 17 13:31:44.145148 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:44.144779    2565 scope.go:117] "RemoveContainer" containerID="1fe75d201e09e6877d05fde2d9bf72758a1ed2534e9e283ad13ea9576e263edb"
Jul 17 13:31:44.145148 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:44.145013    2565 scope.go:117] "RemoveContainer" containerID="8049d20707fc32aaa7147bddb8462e19b99b4ede814b33ea65e6ec2b3e6323ea"
Jul 17 13:31:44.145317 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:31:44.145285    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl_odh-ai-gateway-infra(c878ed16-8d2f-4635-b764-2051883bb31d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl" podUID="c878ed16-8d2f-4635-b764-2051883bb31d"
Jul 17 13:31:44.146582 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:44.146557    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r" event={"ID":"5eae707e-2cb7-47ca-949f-6d38a592f8dc","Type":"ContainerStarted","Data":"68bd303075428a965a7a105c25844d06a2eeb15121df5c9c4bfd9ad66cc4d41c"}
Jul 17 13:31:44.146686 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:44.146596    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r" event={"ID":"5eae707e-2cb7-47ca-949f-6d38a592f8dc","Type":"ContainerStarted","Data":"82a597b28dc2e98a761b949196da23e43eb364650373fa79cc35c3b0bffc0a61"}
Jul 17 13:31:44.179230 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:44.179162    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r" podStartSLOduration=1.179150095 podStartE2EDuration="1.179150095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:31:44.176951922 +0000 UTC m=+4148.308336601" watchObservedRunningTime="2026-07-17 13:31:44.179150095 +0000 UTC m=+4148.310534755"
Jul 17 13:31:45.154088 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:45.154056    2565 generic.go:356] "Generic (PLEG): container finished" podID="5eae707e-2cb7-47ca-949f-6d38a592f8dc" containerID="68bd303075428a965a7a105c25844d06a2eeb15121df5c9c4bfd9ad66cc4d41c" exitCode=6
Jul 17 13:31:45.154564 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:45.154146    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r" event={"ID":"5eae707e-2cb7-47ca-949f-6d38a592f8dc","Type":"ContainerDied","Data":"68bd303075428a965a7a105c25844d06a2eeb15121df5c9c4bfd9ad66cc4d41c"}
Jul 17 13:31:45.154564 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:45.154394    2565 scope.go:117] "RemoveContainer" containerID="68bd303075428a965a7a105c25844d06a2eeb15121df5c9c4bfd9ad66cc4d41c"
Jul 17 13:31:45.155966 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:45.155951    2565 scope.go:117] "RemoveContainer" containerID="8049d20707fc32aaa7147bddb8462e19b99b4ede814b33ea65e6ec2b3e6323ea"
Jul 17 13:31:45.156144 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:31:45.156125    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl_odh-ai-gateway-infra(c878ed16-8d2f-4635-b764-2051883bb31d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl" podUID="c878ed16-8d2f-4635-b764-2051883bb31d"
Jul 17 13:31:46.163898 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:46.163867    2565 generic.go:356] "Generic (PLEG): container finished" podID="5eae707e-2cb7-47ca-949f-6d38a592f8dc" containerID="bee028316b2b1bb159e2157694777cd648aa7649e58d652f5566f83b49f24ec2" exitCode=6
Jul 17 13:31:46.164377 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:46.163936    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r" event={"ID":"5eae707e-2cb7-47ca-949f-6d38a592f8dc","Type":"ContainerDied","Data":"bee028316b2b1bb159e2157694777cd648aa7649e58d652f5566f83b49f24ec2"}
Jul 17 13:31:46.164377 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:46.164008    2565 scope.go:117] "RemoveContainer" containerID="68bd303075428a965a7a105c25844d06a2eeb15121df5c9c4bfd9ad66cc4d41c"
Jul 17 13:31:46.164377 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:46.164193    2565 scope.go:117] "RemoveContainer" containerID="bee028316b2b1bb159e2157694777cd648aa7649e58d652f5566f83b49f24ec2"
Jul 17 13:31:46.164542 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:31:46.164431    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r_odh-ai-gateway-infra(5eae707e-2cb7-47ca-949f-6d38a592f8dc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r" podUID="5eae707e-2cb7-47ca-949f-6d38a592f8dc"
Jul 17 13:31:47.171741 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:47.171713    2565 scope.go:117] "RemoveContainer" containerID="bee028316b2b1bb159e2157694777cd648aa7649e58d652f5566f83b49f24ec2"
Jul 17 13:31:47.172153 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:31:47.171901    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r_odh-ai-gateway-infra(5eae707e-2cb7-47ca-949f-6d38a592f8dc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r" podUID="5eae707e-2cb7-47ca-949f-6d38a592f8dc"
Jul 17 13:31:49.247935 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:49.247894    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8"]
Jul 17 13:31:49.252977 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:49.252960    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8"
Jul 17 13:31:49.263554 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:49.263528    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8"]
Jul 17 13:31:49.421043 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:49.421009    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7a28e7-54b2-4e78-954b-8c27ea5186d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8\" (UID: \"2d7a28e7-54b2-4e78-954b-8c27ea5186d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8"
Jul 17 13:31:49.522230 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:49.522167    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7a28e7-54b2-4e78-954b-8c27ea5186d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8\" (UID: \"2d7a28e7-54b2-4e78-954b-8c27ea5186d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8"
Jul 17 13:31:49.522744 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:49.522726    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7a28e7-54b2-4e78-954b-8c27ea5186d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8\" (UID: \"2d7a28e7-54b2-4e78-954b-8c27ea5186d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8"
Jul 17 13:31:49.564213 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:49.564193    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8"
Jul 17 13:31:49.701756 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:49.701722    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8"]
Jul 17 13:31:49.702798 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:31:49.702776    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7a28e7_54b2_4e78_954b_8c27ea5186d7.slice/crio-fecd98a807f1f917f8252bc2ca80492d2dd932e887635839931412747b7f155d WatchSource:0}: Error finding container fecd98a807f1f917f8252bc2ca80492d2dd932e887635839931412747b7f155d: Status 404 returned error can't find the container with id fecd98a807f1f917f8252bc2ca80492d2dd932e887635839931412747b7f155d
Jul 17 13:31:50.196891 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:50.196816    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8" event={"ID":"2d7a28e7-54b2-4e78-954b-8c27ea5186d7","Type":"ContainerStarted","Data":"0aa081aa2cb1c86a533b7f665906c94ba69b1132cde55e17aa548aa61656bc39"}
Jul 17 13:31:50.196891 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:50.196869    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8" event={"ID":"2d7a28e7-54b2-4e78-954b-8c27ea5186d7","Type":"ContainerStarted","Data":"fecd98a807f1f917f8252bc2ca80492d2dd932e887635839931412747b7f155d"}
Jul 17 13:31:50.216346 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:50.216296    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8" podStartSLOduration=1.216278107 podStartE2EDuration="1.216278107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:31:50.215100712 +0000 UTC m=+4154.346485372" watchObservedRunningTime="2026-07-17 13:31:50.216278107 +0000 UTC m=+4154.347662772"
Jul 17 13:31:51.205144 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:51.205110    2565 generic.go:356] "Generic (PLEG): container finished" podID="2d7a28e7-54b2-4e78-954b-8c27ea5186d7" containerID="0aa081aa2cb1c86a533b7f665906c94ba69b1132cde55e17aa548aa61656bc39" exitCode=6
Jul 17 13:31:51.205574 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:51.205189    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8" event={"ID":"2d7a28e7-54b2-4e78-954b-8c27ea5186d7","Type":"ContainerDied","Data":"0aa081aa2cb1c86a533b7f665906c94ba69b1132cde55e17aa548aa61656bc39"}
Jul 17 13:31:51.205574 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:51.205332    2565 scope.go:117] "RemoveContainer" containerID="0aa081aa2cb1c86a533b7f665906c94ba69b1132cde55e17aa548aa61656bc39"
Jul 17 13:31:52.214192 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:52.214107    2565 generic.go:356] "Generic (PLEG): container finished" podID="2d7a28e7-54b2-4e78-954b-8c27ea5186d7" containerID="9f8acd26a1a112db8f6fe0338bdff655b4c07f5889d81db8d2a14cac09709346" exitCode=6
Jul 17 13:31:52.214595 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:52.214192    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8" event={"ID":"2d7a28e7-54b2-4e78-954b-8c27ea5186d7","Type":"ContainerDied","Data":"9f8acd26a1a112db8f6fe0338bdff655b4c07f5889d81db8d2a14cac09709346"}
Jul 17 13:31:52.214595 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:52.214258    2565 scope.go:117] "RemoveContainer" containerID="0aa081aa2cb1c86a533b7f665906c94ba69b1132cde55e17aa548aa61656bc39"
Jul 17 13:31:52.214595 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:52.214495    2565 scope.go:117] "RemoveContainer" containerID="9f8acd26a1a112db8f6fe0338bdff655b4c07f5889d81db8d2a14cac09709346"
Jul 17 13:31:52.214761 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:31:52.214738    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8_odh-ai-gateway-infra(2d7a28e7-54b2-4e78-954b-8c27ea5186d7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8" podUID="2d7a28e7-54b2-4e78-954b-8c27ea5186d7"
Jul 17 13:31:53.222958 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:53.222927    2565 scope.go:117] "RemoveContainer" containerID="9f8acd26a1a112db8f6fe0338bdff655b4c07f5889d81db8d2a14cac09709346"
Jul 17 13:31:53.223339 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:31:53.223157    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8_odh-ai-gateway-infra(2d7a28e7-54b2-4e78-954b-8c27ea5186d7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8" podUID="2d7a28e7-54b2-4e78-954b-8c27ea5186d7"
Jul 17 13:31:58.396631 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:58.396603    2565 scope.go:117] "RemoveContainer" containerID="8049d20707fc32aaa7147bddb8462e19b99b4ede814b33ea65e6ec2b3e6323ea"
Jul 17 13:31:59.260761 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:59.260727    2565 generic.go:356] "Generic (PLEG): container finished" podID="c878ed16-8d2f-4635-b764-2051883bb31d" containerID="97966cd1adceb15eb3dcef3cdf8961857587071f6f4d74ccfbac3c29e8937773" exitCode=6
Jul 17 13:31:59.260898 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:59.260807    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl" event={"ID":"c878ed16-8d2f-4635-b764-2051883bb31d","Type":"ContainerDied","Data":"97966cd1adceb15eb3dcef3cdf8961857587071f6f4d74ccfbac3c29e8937773"}
Jul 17 13:31:59.260898 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:59.260866    2565 scope.go:117] "RemoveContainer" containerID="8049d20707fc32aaa7147bddb8462e19b99b4ede814b33ea65e6ec2b3e6323ea"
Jul 17 13:31:59.261113 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:59.261090    2565 scope.go:117] "RemoveContainer" containerID="97966cd1adceb15eb3dcef3cdf8961857587071f6f4d74ccfbac3c29e8937773"
Jul 17 13:31:59.261335 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:31:59.261316    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl_odh-ai-gateway-infra(c878ed16-8d2f-4635-b764-2051883bb31d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl" podUID="c878ed16-8d2f-4635-b764-2051883bb31d"
Jul 17 13:31:59.396963 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:31:59.396934    2565 scope.go:117] "RemoveContainer" containerID="bee028316b2b1bb159e2157694777cd648aa7649e58d652f5566f83b49f24ec2"
Jul 17 13:32:00.269071 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:00.268972    2565 generic.go:356] "Generic (PLEG): container finished" podID="5eae707e-2cb7-47ca-949f-6d38a592f8dc" containerID="afa2b3ba30f17d1ee96c9fef36d3f8be693b20c2c7e2a767921ec76f48178c2f" exitCode=6
Jul 17 13:32:00.269071 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:00.269047    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r" event={"ID":"5eae707e-2cb7-47ca-949f-6d38a592f8dc","Type":"ContainerDied","Data":"afa2b3ba30f17d1ee96c9fef36d3f8be693b20c2c7e2a767921ec76f48178c2f"}
Jul 17 13:32:00.269258 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:00.269107    2565 scope.go:117] "RemoveContainer" containerID="bee028316b2b1bb159e2157694777cd648aa7649e58d652f5566f83b49f24ec2"
Jul 17 13:32:00.269392 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:00.269373    2565 scope.go:117] "RemoveContainer" containerID="afa2b3ba30f17d1ee96c9fef36d3f8be693b20c2c7e2a767921ec76f48178c2f"
Jul 17 13:32:00.269614 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:32:00.269586    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r_odh-ai-gateway-infra(5eae707e-2cb7-47ca-949f-6d38a592f8dc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r" podUID="5eae707e-2cb7-47ca-949f-6d38a592f8dc"
Jul 17 13:32:00.291419 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:00.291391    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl"]
Jul 17 13:32:00.435804 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:00.435779    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl"
Jul 17 13:32:00.611942 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:00.611911    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c878ed16-8d2f-4635-b764-2051883bb31d-maas-api-service-ca\") pod \"c878ed16-8d2f-4635-b764-2051883bb31d\" (UID: \"c878ed16-8d2f-4635-b764-2051883bb31d\") "
Jul 17 13:32:00.612248 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:00.612227    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c878ed16-8d2f-4635-b764-2051883bb31d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c878ed16-8d2f-4635-b764-2051883bb31d" (UID: "c878ed16-8d2f-4635-b764-2051883bb31d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:32:00.713424 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:00.713396    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c878ed16-8d2f-4635-b764-2051883bb31d-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:32:01.278764 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:01.278735    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl"
Jul 17 13:32:01.278915 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:01.278787    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl" event={"ID":"c878ed16-8d2f-4635-b764-2051883bb31d","Type":"ContainerDied","Data":"6d0b963d2f830e3f65f2092c4392adff0c39226cdb5fb24be237b160fc281794"}
Jul 17 13:32:01.278915 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:01.278879    2565 scope.go:117] "RemoveContainer" containerID="97966cd1adceb15eb3dcef3cdf8961857587071f6f4d74ccfbac3c29e8937773"
Jul 17 13:32:01.297257 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:01.297236    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r"]
Jul 17 13:32:01.304388 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:01.304366    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl"]
Jul 17 13:32:01.310783 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:01.310763    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-359d73d8-8dwxl"]
Jul 17 13:32:01.447086 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:01.447064    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r"
Jul 17 13:32:01.620727 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:01.620699    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eae707e-2cb7-47ca-949f-6d38a592f8dc-maas-api-service-ca\") pod \"5eae707e-2cb7-47ca-949f-6d38a592f8dc\" (UID: \"5eae707e-2cb7-47ca-949f-6d38a592f8dc\") "
Jul 17 13:32:01.621084 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:01.621060    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eae707e-2cb7-47ca-949f-6d38a592f8dc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5eae707e-2cb7-47ca-949f-6d38a592f8dc" (UID: "5eae707e-2cb7-47ca-949f-6d38a592f8dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:32:01.721917 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:01.721888    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eae707e-2cb7-47ca-949f-6d38a592f8dc-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:32:02.288007 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:02.287961    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r"
Jul 17 13:32:02.288175 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:02.288006    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r" event={"ID":"5eae707e-2cb7-47ca-949f-6d38a592f8dc","Type":"ContainerDied","Data":"82a597b28dc2e98a761b949196da23e43eb364650373fa79cc35c3b0bffc0a61"}
Jul 17 13:32:02.288175 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:02.288068    2565 scope.go:117] "RemoveContainer" containerID="afa2b3ba30f17d1ee96c9fef36d3f8be693b20c2c7e2a767921ec76f48178c2f"
Jul 17 13:32:02.309638 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:02.309609    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r"]
Jul 17 13:32:02.316850 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:02.316814    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3fc8989b-znt9r"]
Jul 17 13:32:02.403028 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:02.402969    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5eae707e-2cb7-47ca-949f-6d38a592f8dc" path="/var/lib/kubelet/pods/5eae707e-2cb7-47ca-949f-6d38a592f8dc/volumes"
Jul 17 13:32:02.403350 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:02.403336    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c878ed16-8d2f-4635-b764-2051883bb31d" path="/var/lib/kubelet/pods/c878ed16-8d2f-4635-b764-2051883bb31d/volumes"
Jul 17 13:32:05.396931 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:05.396902    2565 scope.go:117] "RemoveContainer" containerID="9f8acd26a1a112db8f6fe0338bdff655b4c07f5889d81db8d2a14cac09709346"
Jul 17 13:32:06.322682 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:06.322650    2565 generic.go:356] "Generic (PLEG): container finished" podID="2d7a28e7-54b2-4e78-954b-8c27ea5186d7" containerID="a6585c978760ef570305588f83ad3157dfba27cf65b4bd702175643ba9b7fde2" exitCode=6
Jul 17 13:32:06.322852 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:06.322727    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8" event={"ID":"2d7a28e7-54b2-4e78-954b-8c27ea5186d7","Type":"ContainerDied","Data":"a6585c978760ef570305588f83ad3157dfba27cf65b4bd702175643ba9b7fde2"}
Jul 17 13:32:06.322852 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:06.322825    2565 scope.go:117] "RemoveContainer" containerID="9f8acd26a1a112db8f6fe0338bdff655b4c07f5889d81db8d2a14cac09709346"
Jul 17 13:32:06.323201 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:06.323178    2565 scope.go:117] "RemoveContainer" containerID="a6585c978760ef570305588f83ad3157dfba27cf65b4bd702175643ba9b7fde2"
Jul 17 13:32:06.323463 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:32:06.323443    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8_odh-ai-gateway-infra(2d7a28e7-54b2-4e78-954b-8c27ea5186d7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8" podUID="2d7a28e7-54b2-4e78-954b-8c27ea5186d7"
Jul 17 13:32:07.349287 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:07.349256    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8"]
Jul 17 13:32:07.500788 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:07.500760    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8"
Jul 17 13:32:07.671765 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:07.671685    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7a28e7-54b2-4e78-954b-8c27ea5186d7-maas-api-service-ca\") pod \"2d7a28e7-54b2-4e78-954b-8c27ea5186d7\" (UID: \"2d7a28e7-54b2-4e78-954b-8c27ea5186d7\") "
Jul 17 13:32:07.672026 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:07.671976    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7a28e7-54b2-4e78-954b-8c27ea5186d7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2d7a28e7-54b2-4e78-954b-8c27ea5186d7" (UID: "2d7a28e7-54b2-4e78-954b-8c27ea5186d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:32:07.772323 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:07.772299    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7a28e7-54b2-4e78-954b-8c27ea5186d7-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:32:08.339560 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:08.339518    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8" event={"ID":"2d7a28e7-54b2-4e78-954b-8c27ea5186d7","Type":"ContainerDied","Data":"fecd98a807f1f917f8252bc2ca80492d2dd932e887635839931412747b7f155d"}
Jul 17 13:32:08.339809 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:08.339589    2565 scope.go:117] "RemoveContainer" containerID="a6585c978760ef570305588f83ad3157dfba27cf65b4bd702175643ba9b7fde2"
Jul 17 13:32:08.339809 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:08.339678    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8"
Jul 17 13:32:08.366199 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:08.366176    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8"]
Jul 17 13:32:08.371939 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:08.371920    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-661ee8a2-xrmv8"]
Jul 17 13:32:08.402702 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:08.402681    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d7a28e7-54b2-4e78-954b-8c27ea5186d7" path="/var/lib/kubelet/pods/2d7a28e7-54b2-4e78-954b-8c27ea5186d7/volumes"
Jul 17 13:32:13.604595 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:13.604568    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:32:14.846750 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:14.846712    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:32:15.528617 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.528586    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7684-140520326262320"]
Jul 17 13:32:15.529028 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.529015    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eae707e-2cb7-47ca-949f-6d38a592f8dc" containerName="revoke-keys"
Jul 17 13:32:15.529028 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.529029    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eae707e-2cb7-47ca-949f-6d38a592f8dc" containerName="revoke-keys"
Jul 17 13:32:15.529130 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.529039    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d7a28e7-54b2-4e78-954b-8c27ea5186d7" containerName="revoke-keys"
Jul 17 13:32:15.529130 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.529045    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d7a28e7-54b2-4e78-954b-8c27ea5186d7" containerName="revoke-keys"
Jul 17 13:32:15.529130 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.529058    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c878ed16-8d2f-4635-b764-2051883bb31d" containerName="revoke-keys"
Jul 17 13:32:15.529130 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.529063    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c878ed16-8d2f-4635-b764-2051883bb31d" containerName="revoke-keys"
Jul 17 13:32:15.529130 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.529077    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d7a28e7-54b2-4e78-954b-8c27ea5186d7" containerName="revoke-keys"
Jul 17 13:32:15.529130 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.529081    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d7a28e7-54b2-4e78-954b-8c27ea5186d7" containerName="revoke-keys"
Jul 17 13:32:15.529130 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.529087    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eae707e-2cb7-47ca-949f-6d38a592f8dc" containerName="revoke-keys"
Jul 17 13:32:15.529130 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.529092    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eae707e-2cb7-47ca-949f-6d38a592f8dc" containerName="revoke-keys"
Jul 17 13:32:15.529130 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.529099    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c878ed16-8d2f-4635-b764-2051883bb31d" containerName="revoke-keys"
Jul 17 13:32:15.529130 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.529104    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c878ed16-8d2f-4635-b764-2051883bb31d" containerName="revoke-keys"
Jul 17 13:32:15.529532 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.529169    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eae707e-2cb7-47ca-949f-6d38a592f8dc" containerName="revoke-keys"
Jul 17 13:32:15.529532 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.529176    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d7a28e7-54b2-4e78-954b-8c27ea5186d7" containerName="revoke-keys"
Jul 17 13:32:15.529532 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.529180    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c878ed16-8d2f-4635-b764-2051883bb31d" containerName="revoke-keys"
Jul 17 13:32:15.529532 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.529185    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d7a28e7-54b2-4e78-954b-8c27ea5186d7" containerName="revoke-keys"
Jul 17 13:32:15.529532 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.529190    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eae707e-2cb7-47ca-949f-6d38a592f8dc" containerName="revoke-keys"
Jul 17 13:32:15.529532 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.529196    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c878ed16-8d2f-4635-b764-2051883bb31d" containerName="revoke-keys"
Jul 17 13:32:15.529532 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.529203    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c878ed16-8d2f-4635-b764-2051883bb31d" containerName="revoke-keys"
Jul 17 13:32:15.533599 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.533571    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326262320"
Jul 17 13:32:15.535680 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.535656    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-140520326262320"]
Jul 17 13:32:15.536440 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.536417    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cqmd5\""
Jul 17 13:32:15.642078 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.642047    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq2jk\" (UniqueName: \"kubernetes.io/projected/8c513da7-6862-40c8-b402-cdad7baa24bc-kube-api-access-rq2jk\") pod \"test-curl-7684-140520326262320\" (UID: \"8c513da7-6862-40c8-b402-cdad7baa24bc\") " pod="opendatahub/test-curl-7684-140520326262320"
Jul 17 13:32:15.742744 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.742713    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq2jk\" (UniqueName: \"kubernetes.io/projected/8c513da7-6862-40c8-b402-cdad7baa24bc-kube-api-access-rq2jk\") pod \"test-curl-7684-140520326262320\" (UID: \"8c513da7-6862-40c8-b402-cdad7baa24bc\") " pod="opendatahub/test-curl-7684-140520326262320"
Jul 17 13:32:15.752128 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.752102    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq2jk\" (UniqueName: \"kubernetes.io/projected/8c513da7-6862-40c8-b402-cdad7baa24bc-kube-api-access-rq2jk\") pod \"test-curl-7684-140520326262320\" (UID: \"8c513da7-6862-40c8-b402-cdad7baa24bc\") " pod="opendatahub/test-curl-7684-140520326262320"
Jul 17 13:32:15.844196 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.844168    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326262320"
Jul 17 13:32:15.990620 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:32:15.990593    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c513da7_6862_40c8_b402_cdad7baa24bc.slice/crio-3d465900d6a7b0c5b4261ebbe6b216b010f53b6cbe7d51f32ad01124586b6dd7 WatchSource:0}: Error finding container 3d465900d6a7b0c5b4261ebbe6b216b010f53b6cbe7d51f32ad01124586b6dd7: Status 404 returned error can't find the container with id 3d465900d6a7b0c5b4261ebbe6b216b010f53b6cbe7d51f32ad01124586b6dd7
Jul 17 13:32:15.991022 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:15.990946    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-140520326262320"]
Jul 17 13:32:16.021856 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:16.021831    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:32:16.121796 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:16.121720    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:32:16.392151 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:16.392060    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-140520326262320" event={"ID":"8c513da7-6862-40c8-b402-cdad7baa24bc","Type":"ContainerStarted","Data":"3d465900d6a7b0c5b4261ebbe6b216b010f53b6cbe7d51f32ad01124586b6dd7"}
Jul 17 13:32:17.400279 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:17.400241    2565 generic.go:356] "Generic (PLEG): container finished" podID="8c513da7-6862-40c8-b402-cdad7baa24bc" containerID="b128d7e43d654ab0df37a181fd71c8e9c9334c7aaff9ad71d6d5486fbc3424a1" exitCode=0
Jul 17 13:32:17.400640 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:17.400322    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-140520326262320" event={"ID":"8c513da7-6862-40c8-b402-cdad7baa24bc","Type":"ContainerDied","Data":"b128d7e43d654ab0df37a181fd71c8e9c9334c7aaff9ad71d6d5486fbc3424a1"}
Jul 17 13:32:18.543061 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.543039    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326262320"
Jul 17 13:32:18.564919 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.564892    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7684-140520326262320_8c513da7-6862-40c8-b402-cdad7baa24bc/test-curl-7684-140520326262320/0.log"
Jul 17 13:32:18.568058 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.568040    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rq2jk\" (UniqueName: \"kubernetes.io/projected/8c513da7-6862-40c8-b402-cdad7baa24bc-kube-api-access-rq2jk\") pod \"8c513da7-6862-40c8-b402-cdad7baa24bc\" (UID: \"8c513da7-6862-40c8-b402-cdad7baa24bc\") "
Jul 17 13:32:18.570209 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.570186    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c513da7-6862-40c8-b402-cdad7baa24bc-kube-api-access-rq2jk" (OuterVolumeSpecName: "kube-api-access-rq2jk") pod "8c513da7-6862-40c8-b402-cdad7baa24bc" (UID: "8c513da7-6862-40c8-b402-cdad7baa24bc"). InnerVolumeSpecName "kube-api-access-rq2jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:32:18.599542 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.599507    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7684-140520326262320"]
Jul 17 13:32:18.602127 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.602107    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7684-140520326262320"]
Jul 17 13:32:18.669409 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.669332    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq2jk\" (UniqueName: \"kubernetes.io/projected/8c513da7-6862-40c8-b402-cdad7baa24bc-kube-api-access-rq2jk\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:32:18.719310 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.719287    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7684-140520326226608"]
Jul 17 13:32:18.719766 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.719753    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d7a28e7-54b2-4e78-954b-8c27ea5186d7" containerName="revoke-keys"
Jul 17 13:32:18.719814 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.719769    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d7a28e7-54b2-4e78-954b-8c27ea5186d7" containerName="revoke-keys"
Jul 17 13:32:18.719814 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.719785    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c878ed16-8d2f-4635-b764-2051883bb31d" containerName="revoke-keys"
Jul 17 13:32:18.719814 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.719791    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c878ed16-8d2f-4635-b764-2051883bb31d" containerName="revoke-keys"
Jul 17 13:32:18.719814 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.719808    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eae707e-2cb7-47ca-949f-6d38a592f8dc" containerName="revoke-keys"
Jul 17 13:32:18.719931 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.719817    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eae707e-2cb7-47ca-949f-6d38a592f8dc" containerName="revoke-keys"
Jul 17 13:32:18.719931 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.719832    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c513da7-6862-40c8-b402-cdad7baa24bc" containerName="test-curl-7684-140520326262320"
Jul 17 13:32:18.719931 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.719838    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c513da7-6862-40c8-b402-cdad7baa24bc" containerName="test-curl-7684-140520326262320"
Jul 17 13:32:18.719931 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.719906    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c513da7-6862-40c8-b402-cdad7baa24bc" containerName="test-curl-7684-140520326262320"
Jul 17 13:32:18.719931 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.719916    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eae707e-2cb7-47ca-949f-6d38a592f8dc" containerName="revoke-keys"
Jul 17 13:32:18.719931 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.719922    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d7a28e7-54b2-4e78-954b-8c27ea5186d7" containerName="revoke-keys"
Jul 17 13:32:18.722937 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.722921    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326226608"
Jul 17 13:32:18.727571 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.727550    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-140520326226608"]
Jul 17 13:32:18.770689 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.770663    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8tfb\" (UniqueName: \"kubernetes.io/projected/9c082adb-da55-46ea-82c8-77f4f65c6642-kube-api-access-f8tfb\") pod \"test-curl-7684-140520326226608\" (UID: \"9c082adb-da55-46ea-82c8-77f4f65c6642\") " pod="opendatahub/test-curl-7684-140520326226608"
Jul 17 13:32:18.871666 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.871626    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8tfb\" (UniqueName: \"kubernetes.io/projected/9c082adb-da55-46ea-82c8-77f4f65c6642-kube-api-access-f8tfb\") pod \"test-curl-7684-140520326226608\" (UID: \"9c082adb-da55-46ea-82c8-77f4f65c6642\") " pod="opendatahub/test-curl-7684-140520326226608"
Jul 17 13:32:18.880820 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:18.880790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8tfb\" (UniqueName: \"kubernetes.io/projected/9c082adb-da55-46ea-82c8-77f4f65c6642-kube-api-access-f8tfb\") pod \"test-curl-7684-140520326226608\" (UID: \"9c082adb-da55-46ea-82c8-77f4f65c6642\") " pod="opendatahub/test-curl-7684-140520326226608"
Jul 17 13:32:19.034215 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:19.034138    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326226608"
Jul 17 13:32:19.175353 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:32:19.175319    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c082adb_da55_46ea_82c8_77f4f65c6642.slice/crio-e5675bdc5bb0241f5aa4498dd556e355f9bf4c692e5526e8633514e585d0536b WatchSource:0}: Error finding container e5675bdc5bb0241f5aa4498dd556e355f9bf4c692e5526e8633514e585d0536b: Status 404 returned error can't find the container with id e5675bdc5bb0241f5aa4498dd556e355f9bf4c692e5526e8633514e585d0536b
Jul 17 13:32:19.175715 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:19.175602    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-140520326226608"]
Jul 17 13:32:19.413914 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:19.413880    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-140520326226608" event={"ID":"9c082adb-da55-46ea-82c8-77f4f65c6642","Type":"ContainerStarted","Data":"e5675bdc5bb0241f5aa4498dd556e355f9bf4c692e5526e8633514e585d0536b"}
Jul 17 13:32:19.415250 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:19.415229    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326262320"
Jul 17 13:32:19.415375 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:19.415268    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d465900d6a7b0c5b4261ebbe6b216b010f53b6cbe7d51f32ad01124586b6dd7"
Jul 17 13:32:19.425716 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:32:19.425687    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7684-140520326262320\" is forbidden: User \"system:node:ip-10-0-134-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-119.ec2.internal' and this object" podUID="8c513da7-6862-40c8-b402-cdad7baa24bc" pod="opendatahub/test-curl-7684-140520326262320"
Jul 17 13:32:20.403324 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:20.403291    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c513da7-6862-40c8-b402-cdad7baa24bc" path="/var/lib/kubelet/pods/8c513da7-6862-40c8-b402-cdad7baa24bc/volumes"
Jul 17 13:32:20.423181 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:20.423145    2565 generic.go:356] "Generic (PLEG): container finished" podID="9c082adb-da55-46ea-82c8-77f4f65c6642" containerID="635fa624f21407ea696f6ddb57bd972cac294a921a62b7e7a1771fb3d2bf8aeb" exitCode=0
Jul 17 13:32:20.423341 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:20.423215    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-140520326226608" event={"ID":"9c082adb-da55-46ea-82c8-77f4f65c6642","Type":"ContainerDied","Data":"635fa624f21407ea696f6ddb57bd972cac294a921a62b7e7a1771fb3d2bf8aeb"}
Jul 17 13:32:20.580268 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:20.580237    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr"]
Jul 17 13:32:20.583905 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:20.583890    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr"
Jul 17 13:32:20.586339 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:20.586320    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:32:20.591597 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:20.591569    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr"]
Jul 17 13:32:20.689021 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:20.688923    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8201c6fb-124c-4c58-a36f-0b036a8f2efb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr\" (UID: \"8201c6fb-124c-4c58-a36f-0b036a8f2efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr"
Jul 17 13:32:20.790393 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:20.790365    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8201c6fb-124c-4c58-a36f-0b036a8f2efb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr\" (UID: \"8201c6fb-124c-4c58-a36f-0b036a8f2efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr"
Jul 17 13:32:20.790928 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:20.790909    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8201c6fb-124c-4c58-a36f-0b036a8f2efb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr\" (UID: \"8201c6fb-124c-4c58-a36f-0b036a8f2efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr"
Jul 17 13:32:20.896089 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:20.896049    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr"
Jul 17 13:32:21.031272 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:21.031239    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr"]
Jul 17 13:32:21.032352 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:32:21.032321    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8201c6fb_124c_4c58_a36f_0b036a8f2efb.slice/crio-3424c97a5c272709a556c7636804ab8760b957bca893636b7219aa5d101255a1 WatchSource:0}: Error finding container 3424c97a5c272709a556c7636804ab8760b957bca893636b7219aa5d101255a1: Status 404 returned error can't find the container with id 3424c97a5c272709a556c7636804ab8760b957bca893636b7219aa5d101255a1
Jul 17 13:32:21.431678 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:21.431645    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr" event={"ID":"8201c6fb-124c-4c58-a36f-0b036a8f2efb","Type":"ContainerStarted","Data":"f865a7bad42bd01943d32fa69089415c3cad167dac0d25a1fe660199226e64e1"}
Jul 17 13:32:21.432068 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:21.431702    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr" event={"ID":"8201c6fb-124c-4c58-a36f-0b036a8f2efb","Type":"ContainerStarted","Data":"3424c97a5c272709a556c7636804ab8760b957bca893636b7219aa5d101255a1"}
Jul 17 13:32:21.448474 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:21.448428    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr" podStartSLOduration=1.448413106 podStartE2EDuration="1.448413106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:32:21.447928702 +0000 UTC m=+4185.579313363" watchObservedRunningTime="2026-07-17 13:32:21.448413106 +0000 UTC m=+4185.579797766"
Jul 17 13:32:21.566013 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:21.565978    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326226608"
Jul 17 13:32:21.588169 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:21.588140    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7684-140520326226608_9c082adb-da55-46ea-82c8-77f4f65c6642/test-curl-7684-140520326226608/0.log"
Jul 17 13:32:21.597475 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:21.597458    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8tfb\" (UniqueName: \"kubernetes.io/projected/9c082adb-da55-46ea-82c8-77f4f65c6642-kube-api-access-f8tfb\") pod \"9c082adb-da55-46ea-82c8-77f4f65c6642\" (UID: \"9c082adb-da55-46ea-82c8-77f4f65c6642\") "
Jul 17 13:32:21.599512 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:21.599493    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c082adb-da55-46ea-82c8-77f4f65c6642-kube-api-access-f8tfb" (OuterVolumeSpecName: "kube-api-access-f8tfb") pod "9c082adb-da55-46ea-82c8-77f4f65c6642" (UID: "9c082adb-da55-46ea-82c8-77f4f65c6642"). InnerVolumeSpecName "kube-api-access-f8tfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:32:21.614675 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:21.614649    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7684-140520326226608"]
Jul 17 13:32:21.618593 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:21.618569    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7684-140520326226608"]
Jul 17 13:32:21.698220 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:21.698191    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8tfb\" (UniqueName: \"kubernetes.io/projected/9c082adb-da55-46ea-82c8-77f4f65c6642-kube-api-access-f8tfb\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:32:21.783692 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:21.783632    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7684-140520326264848"]
Jul 17 13:32:21.784089 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:21.784075    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c082adb-da55-46ea-82c8-77f4f65c6642" containerName="test-curl-7684-140520326226608"
Jul 17 13:32:21.784133 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:21.784090    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c082adb-da55-46ea-82c8-77f4f65c6642" containerName="test-curl-7684-140520326226608"
Jul 17 13:32:21.784176 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:21.784169    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c082adb-da55-46ea-82c8-77f4f65c6642" containerName="test-curl-7684-140520326226608"
Jul 17 13:32:21.787438 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:21.787423    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326264848"
Jul 17 13:32:21.794803 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:21.794776    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-140520326264848"]
Jul 17 13:32:21.900070 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:21.900039    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w54cn\" (UniqueName: \"kubernetes.io/projected/6f88a410-fbff-4005-8de2-7922b7fdaa04-kube-api-access-w54cn\") pod \"test-curl-7684-140520326264848\" (UID: \"6f88a410-fbff-4005-8de2-7922b7fdaa04\") " pod="opendatahub/test-curl-7684-140520326264848"
Jul 17 13:32:22.001223 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:22.001189    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w54cn\" (UniqueName: \"kubernetes.io/projected/6f88a410-fbff-4005-8de2-7922b7fdaa04-kube-api-access-w54cn\") pod \"test-curl-7684-140520326264848\" (UID: \"6f88a410-fbff-4005-8de2-7922b7fdaa04\") " pod="opendatahub/test-curl-7684-140520326264848"
Jul 17 13:32:22.010263 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:22.010241    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w54cn\" (UniqueName: \"kubernetes.io/projected/6f88a410-fbff-4005-8de2-7922b7fdaa04-kube-api-access-w54cn\") pod \"test-curl-7684-140520326264848\" (UID: \"6f88a410-fbff-4005-8de2-7922b7fdaa04\") " pod="opendatahub/test-curl-7684-140520326264848"
Jul 17 13:32:22.098778 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:22.098750    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326264848"
Jul 17 13:32:22.235710 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:22.235680    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-140520326264848"]
Jul 17 13:32:22.236700 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:32:22.236669    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f88a410_fbff_4005_8de2_7922b7fdaa04.slice/crio-0f8350afb4104d6d93664c5098aa826841d2d4df3cd5dadd47c8b53d6dd4e306 WatchSource:0}: Error finding container 0f8350afb4104d6d93664c5098aa826841d2d4df3cd5dadd47c8b53d6dd4e306: Status 404 returned error can't find the container with id 0f8350afb4104d6d93664c5098aa826841d2d4df3cd5dadd47c8b53d6dd4e306
Jul 17 13:32:22.404137 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:22.404044    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c082adb-da55-46ea-82c8-77f4f65c6642" path="/var/lib/kubelet/pods/9c082adb-da55-46ea-82c8-77f4f65c6642/volumes"
Jul 17 13:32:22.439230 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:22.439199    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326226608"
Jul 17 13:32:22.439643 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:22.439198    2565 scope.go:117] "RemoveContainer" containerID="635fa624f21407ea696f6ddb57bd972cac294a921a62b7e7a1771fb3d2bf8aeb"
Jul 17 13:32:22.441085 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:22.441055    2565 generic.go:356] "Generic (PLEG): container finished" podID="8201c6fb-124c-4c58-a36f-0b036a8f2efb" containerID="f865a7bad42bd01943d32fa69089415c3cad167dac0d25a1fe660199226e64e1" exitCode=6
Jul 17 13:32:22.441235 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:22.441162    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr" event={"ID":"8201c6fb-124c-4c58-a36f-0b036a8f2efb","Type":"ContainerDied","Data":"f865a7bad42bd01943d32fa69089415c3cad167dac0d25a1fe660199226e64e1"}
Jul 17 13:32:22.441472 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:22.441318    2565 scope.go:117] "RemoveContainer" containerID="f865a7bad42bd01943d32fa69089415c3cad167dac0d25a1fe660199226e64e1"
Jul 17 13:32:22.442817 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:22.442784    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-140520326264848" event={"ID":"6f88a410-fbff-4005-8de2-7922b7fdaa04","Type":"ContainerStarted","Data":"0f8350afb4104d6d93664c5098aa826841d2d4df3cd5dadd47c8b53d6dd4e306"}
Jul 17 13:32:23.451186 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:23.451102    2565 generic.go:356] "Generic (PLEG): container finished" podID="6f88a410-fbff-4005-8de2-7922b7fdaa04" containerID="308a0b4ff2b6fe9c74f7fdf88c38cf7969ea9db36fe2966294fe881ae402211b" exitCode=0
Jul 17 13:32:23.451561 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:23.451192    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-140520326264848" event={"ID":"6f88a410-fbff-4005-8de2-7922b7fdaa04","Type":"ContainerDied","Data":"308a0b4ff2b6fe9c74f7fdf88c38cf7969ea9db36fe2966294fe881ae402211b"}
Jul 17 13:32:23.454003 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:23.453963    2565 generic.go:356] "Generic (PLEG): container finished" podID="8201c6fb-124c-4c58-a36f-0b036a8f2efb" containerID="151816836cb4f8efa89fa7ed97f32592f6afb1018cb00b558b5c70cbcc2afb9d" exitCode=6
Jul 17 13:32:23.454140 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:23.454011    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr" event={"ID":"8201c6fb-124c-4c58-a36f-0b036a8f2efb","Type":"ContainerDied","Data":"151816836cb4f8efa89fa7ed97f32592f6afb1018cb00b558b5c70cbcc2afb9d"}
Jul 17 13:32:23.454140 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:23.454063    2565 scope.go:117] "RemoveContainer" containerID="f865a7bad42bd01943d32fa69089415c3cad167dac0d25a1fe660199226e64e1"
Jul 17 13:32:23.454278 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:23.454260    2565 scope.go:117] "RemoveContainer" containerID="151816836cb4f8efa89fa7ed97f32592f6afb1018cb00b558b5c70cbcc2afb9d"
Jul 17 13:32:23.454530 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:32:23.454508    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr_odh-ai-gateway-infra(8201c6fb-124c-4c58-a36f-0b036a8f2efb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr" podUID="8201c6fb-124c-4c58-a36f-0b036a8f2efb"
Jul 17 13:32:24.081868 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:24.081839    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:32:24.182131 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:24.182084    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:32:24.461891 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:24.461796    2565 scope.go:117] "RemoveContainer" containerID="151816836cb4f8efa89fa7ed97f32592f6afb1018cb00b558b5c70cbcc2afb9d"
Jul 17 13:32:24.462624 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:32:24.462030    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr_odh-ai-gateway-infra(8201c6fb-124c-4c58-a36f-0b036a8f2efb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr" podUID="8201c6fb-124c-4c58-a36f-0b036a8f2efb"
Jul 17 13:32:24.608506 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:24.608481    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326264848"
Jul 17 13:32:24.629033 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:24.629009    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7684-140520326264848_6f88a410-fbff-4005-8de2-7922b7fdaa04/test-curl-7684-140520326264848/0.log"
Jul 17 13:32:24.655942 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:24.655912    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7684-140520326264848"]
Jul 17 13:32:24.659027 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:24.659006    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7684-140520326264848"]
Jul 17 13:32:24.727040 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:24.726944    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w54cn\" (UniqueName: \"kubernetes.io/projected/6f88a410-fbff-4005-8de2-7922b7fdaa04-kube-api-access-w54cn\") pod \"6f88a410-fbff-4005-8de2-7922b7fdaa04\" (UID: \"6f88a410-fbff-4005-8de2-7922b7fdaa04\") "
Jul 17 13:32:24.729073 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:24.729041    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f88a410-fbff-4005-8de2-7922b7fdaa04-kube-api-access-w54cn" (OuterVolumeSpecName: "kube-api-access-w54cn") pod "6f88a410-fbff-4005-8de2-7922b7fdaa04" (UID: "6f88a410-fbff-4005-8de2-7922b7fdaa04"). InnerVolumeSpecName "kube-api-access-w54cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:32:24.775631 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:24.775602    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7684-140520326264848"]
Jul 17 13:32:24.776083 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:24.776067    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f88a410-fbff-4005-8de2-7922b7fdaa04" containerName="test-curl-7684-140520326264848"
Jul 17 13:32:24.776083 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:24.776083    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f88a410-fbff-4005-8de2-7922b7fdaa04" containerName="test-curl-7684-140520326264848"
Jul 17 13:32:24.776178 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:24.776169    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f88a410-fbff-4005-8de2-7922b7fdaa04" containerName="test-curl-7684-140520326264848"
Jul 17 13:32:24.779254 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:24.779239    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326264848"
Jul 17 13:32:24.785065 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:24.785039    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-140520326264848"]
Jul 17 13:32:24.828061 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:24.828033    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-966mv\" (UniqueName: \"kubernetes.io/projected/294e1136-a6f2-4078-adb7-f3c1905e27df-kube-api-access-966mv\") pod \"test-curl-7684-140520326264848\" (UID: \"294e1136-a6f2-4078-adb7-f3c1905e27df\") " pod="opendatahub/test-curl-7684-140520326264848"
Jul 17 13:32:24.828239 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:24.828216    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w54cn\" (UniqueName: \"kubernetes.io/projected/6f88a410-fbff-4005-8de2-7922b7fdaa04-kube-api-access-w54cn\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:32:24.929572 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:24.929542    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-966mv\" (UniqueName: \"kubernetes.io/projected/294e1136-a6f2-4078-adb7-f3c1905e27df-kube-api-access-966mv\") pod \"test-curl-7684-140520326264848\" (UID: \"294e1136-a6f2-4078-adb7-f3c1905e27df\") " pod="opendatahub/test-curl-7684-140520326264848"
Jul 17 13:32:24.937595 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:24.937570    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-966mv\" (UniqueName: \"kubernetes.io/projected/294e1136-a6f2-4078-adb7-f3c1905e27df-kube-api-access-966mv\") pod \"test-curl-7684-140520326264848\" (UID: \"294e1136-a6f2-4078-adb7-f3c1905e27df\") " pod="opendatahub/test-curl-7684-140520326264848"
Jul 17 13:32:25.090228 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:25.090193    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326264848"
Jul 17 13:32:25.246394 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:25.246370    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-140520326264848"]
Jul 17 13:32:25.248215 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:32:25.248187    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod294e1136_a6f2_4078_adb7_f3c1905e27df.slice/crio-19649dd23ac9bc28a8b3ee97ab731d59d2623a55db6587666432bacc450f1b55 WatchSource:0}: Error finding container 19649dd23ac9bc28a8b3ee97ab731d59d2623a55db6587666432bacc450f1b55: Status 404 returned error can't find the container with id 19649dd23ac9bc28a8b3ee97ab731d59d2623a55db6587666432bacc450f1b55
Jul 17 13:32:25.469247 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:25.469156    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-140520326264848" event={"ID":"294e1136-a6f2-4078-adb7-f3c1905e27df","Type":"ContainerStarted","Data":"19649dd23ac9bc28a8b3ee97ab731d59d2623a55db6587666432bacc450f1b55"}
Jul 17 13:32:25.470508 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:25.470487    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f8350afb4104d6d93664c5098aa826841d2d4df3cd5dadd47c8b53d6dd4e306"
Jul 17 13:32:25.470606 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:25.470543    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326264848"
Jul 17 13:32:25.482688 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:25.482656    2565 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7684-140520326264848" oldPodUID="6f88a410-fbff-4005-8de2-7922b7fdaa04" podUID="294e1136-a6f2-4078-adb7-f3c1905e27df"
Jul 17 13:32:26.403846 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:26.403816    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f88a410-fbff-4005-8de2-7922b7fdaa04" path="/var/lib/kubelet/pods/6f88a410-fbff-4005-8de2-7922b7fdaa04/volumes"
Jul 17 13:32:26.477752 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:26.477722    2565 generic.go:356] "Generic (PLEG): container finished" podID="294e1136-a6f2-4078-adb7-f3c1905e27df" containerID="3f55492d49fce56e5590cb304c429d5ab8f55d14d6fa7cdf367135886a052da9" exitCode=0
Jul 17 13:32:26.478133 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:26.477806    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-140520326264848" event={"ID":"294e1136-a6f2-4078-adb7-f3c1905e27df","Type":"ContainerDied","Data":"3f55492d49fce56e5590cb304c429d5ab8f55d14d6fa7cdf367135886a052da9"}
Jul 17 13:32:27.624773 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:27.624749    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326264848"
Jul 17 13:32:27.645343 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:27.645319    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7684-140520326264848_294e1136-a6f2-4078-adb7-f3c1905e27df/test-curl-7684-140520326264848/0.log"
Jul 17 13:32:27.672304 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:27.672274    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7684-140520326264848"]
Jul 17 13:32:27.675675 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:27.675653    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7684-140520326264848"]
Jul 17 13:32:27.754466 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:27.754408    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-966mv\" (UniqueName: \"kubernetes.io/projected/294e1136-a6f2-4078-adb7-f3c1905e27df-kube-api-access-966mv\") pod \"294e1136-a6f2-4078-adb7-f3c1905e27df\" (UID: \"294e1136-a6f2-4078-adb7-f3c1905e27df\") "
Jul 17 13:32:27.756606 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:27.756582    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294e1136-a6f2-4078-adb7-f3c1905e27df-kube-api-access-966mv" (OuterVolumeSpecName: "kube-api-access-966mv") pod "294e1136-a6f2-4078-adb7-f3c1905e27df" (UID: "294e1136-a6f2-4078-adb7-f3c1905e27df"). InnerVolumeSpecName "kube-api-access-966mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:32:27.804446 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:27.804419    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7684-140520326265856"]
Jul 17 13:32:27.804912 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:27.804885    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="294e1136-a6f2-4078-adb7-f3c1905e27df" containerName="test-curl-7684-140520326264848"
Jul 17 13:32:27.804912 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:27.804907    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="294e1136-a6f2-4078-adb7-f3c1905e27df" containerName="test-curl-7684-140520326264848"
Jul 17 13:32:27.805083 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:27.805072    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="294e1136-a6f2-4078-adb7-f3c1905e27df" containerName="test-curl-7684-140520326264848"
Jul 17 13:32:27.808261 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:27.808244    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326265856"
Jul 17 13:32:27.815818 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:27.815792    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-140520326265856"]
Jul 17 13:32:27.855508 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:27.855487    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c67md\" (UniqueName: \"kubernetes.io/projected/4006fab5-7995-4072-826d-b5afa535ad0e-kube-api-access-c67md\") pod \"test-curl-7684-140520326265856\" (UID: \"4006fab5-7995-4072-826d-b5afa535ad0e\") " pod="opendatahub/test-curl-7684-140520326265856"
Jul 17 13:32:27.855602 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:27.855559    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-966mv\" (UniqueName: \"kubernetes.io/projected/294e1136-a6f2-4078-adb7-f3c1905e27df-kube-api-access-966mv\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:32:27.956521 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:27.956492    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c67md\" (UniqueName: \"kubernetes.io/projected/4006fab5-7995-4072-826d-b5afa535ad0e-kube-api-access-c67md\") pod \"test-curl-7684-140520326265856\" (UID: \"4006fab5-7995-4072-826d-b5afa535ad0e\") " pod="opendatahub/test-curl-7684-140520326265856"
Jul 17 13:32:27.964540 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:27.964519    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c67md\" (UniqueName: \"kubernetes.io/projected/4006fab5-7995-4072-826d-b5afa535ad0e-kube-api-access-c67md\") pod \"test-curl-7684-140520326265856\" (UID: \"4006fab5-7995-4072-826d-b5afa535ad0e\") " pod="opendatahub/test-curl-7684-140520326265856"
Jul 17 13:32:28.119469 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:28.119445    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326265856"
Jul 17 13:32:28.257282 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:28.257254    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-140520326265856"]
Jul 17 13:32:28.258109 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:32:28.258083    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4006fab5_7995_4072_826d_b5afa535ad0e.slice/crio-91024515c1ee523e6bf1b0875a9b330043907b5ff109960304c76c8a4d083188 WatchSource:0}: Error finding container 91024515c1ee523e6bf1b0875a9b330043907b5ff109960304c76c8a4d083188: Status 404 returned error can't find the container with id 91024515c1ee523e6bf1b0875a9b330043907b5ff109960304c76c8a4d083188
Jul 17 13:32:28.403749 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:28.403691    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="294e1136-a6f2-4078-adb7-f3c1905e27df" path="/var/lib/kubelet/pods/294e1136-a6f2-4078-adb7-f3c1905e27df/volumes"
Jul 17 13:32:28.491578 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:28.491551    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326264848"
Jul 17 13:32:28.491697 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:28.491598    2565 scope.go:117] "RemoveContainer" containerID="3f55492d49fce56e5590cb304c429d5ab8f55d14d6fa7cdf367135886a052da9"
Jul 17 13:32:28.492952 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:28.492927    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-140520326265856" event={"ID":"4006fab5-7995-4072-826d-b5afa535ad0e","Type":"ContainerStarted","Data":"91024515c1ee523e6bf1b0875a9b330043907b5ff109960304c76c8a4d083188"}
Jul 17 13:32:29.500351 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:29.500321    2565 generic.go:356] "Generic (PLEG): container finished" podID="4006fab5-7995-4072-826d-b5afa535ad0e" containerID="a1bed96affcce3fb90808d793d815eb2d40092e89108dfa04d0dc55d3e82f99a" exitCode=0
Jul 17 13:32:29.500736 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:29.500403    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-140520326265856" event={"ID":"4006fab5-7995-4072-826d-b5afa535ad0e","Type":"ContainerDied","Data":"a1bed96affcce3fb90808d793d815eb2d40092e89108dfa04d0dc55d3e82f99a"}
Jul 17 13:32:30.649781 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:30.649758    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326265856"
Jul 17 13:32:30.669915 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:30.669852    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7684-140520326265856_4006fab5-7995-4072-826d-b5afa535ad0e/test-curl-7684-140520326265856/0.log"
Jul 17 13:32:30.695284 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:30.695258    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7684-140520326265856"]
Jul 17 13:32:30.699173 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:30.699155    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7684-140520326265856"]
Jul 17 13:32:30.780401 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:30.780321    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c67md\" (UniqueName: \"kubernetes.io/projected/4006fab5-7995-4072-826d-b5afa535ad0e-kube-api-access-c67md\") pod \"4006fab5-7995-4072-826d-b5afa535ad0e\" (UID: \"4006fab5-7995-4072-826d-b5afa535ad0e\") "
Jul 17 13:32:30.782465 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:30.782436    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4006fab5-7995-4072-826d-b5afa535ad0e-kube-api-access-c67md" (OuterVolumeSpecName: "kube-api-access-c67md") pod "4006fab5-7995-4072-826d-b5afa535ad0e" (UID: "4006fab5-7995-4072-826d-b5afa535ad0e"). InnerVolumeSpecName "kube-api-access-c67md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:32:30.819434 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:30.819407    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7684-140520326265856"]
Jul 17 13:32:30.819837 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:30.819824    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4006fab5-7995-4072-826d-b5afa535ad0e" containerName="test-curl-7684-140520326265856"
Jul 17 13:32:30.819888 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:30.819840    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4006fab5-7995-4072-826d-b5afa535ad0e" containerName="test-curl-7684-140520326265856"
Jul 17 13:32:30.819946 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:30.819937    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4006fab5-7995-4072-826d-b5afa535ad0e" containerName="test-curl-7684-140520326265856"
Jul 17 13:32:30.823088 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:30.823072    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326265856"
Jul 17 13:32:30.857142 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:30.857113    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-140520326265856"]
Jul 17 13:32:30.881476 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:30.881439    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpz8v\" (UniqueName: \"kubernetes.io/projected/bee8a3b9-5e3a-4629-ae9d-3140fafbf420-kube-api-access-vpz8v\") pod \"test-curl-7684-140520326265856\" (UID: \"bee8a3b9-5e3a-4629-ae9d-3140fafbf420\") " pod="opendatahub/test-curl-7684-140520326265856"
Jul 17 13:32:30.881607 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:30.881538    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c67md\" (UniqueName: \"kubernetes.io/projected/4006fab5-7995-4072-826d-b5afa535ad0e-kube-api-access-c67md\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:32:30.982292 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:30.982264    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpz8v\" (UniqueName: \"kubernetes.io/projected/bee8a3b9-5e3a-4629-ae9d-3140fafbf420-kube-api-access-vpz8v\") pod \"test-curl-7684-140520326265856\" (UID: \"bee8a3b9-5e3a-4629-ae9d-3140fafbf420\") " pod="opendatahub/test-curl-7684-140520326265856"
Jul 17 13:32:30.991602 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:30.991580    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpz8v\" (UniqueName: \"kubernetes.io/projected/bee8a3b9-5e3a-4629-ae9d-3140fafbf420-kube-api-access-vpz8v\") pod \"test-curl-7684-140520326265856\" (UID: \"bee8a3b9-5e3a-4629-ae9d-3140fafbf420\") " pod="opendatahub/test-curl-7684-140520326265856"
Jul 17 13:32:31.135190 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:31.135167    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326265856"
Jul 17 13:32:31.274218 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:31.274192    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-140520326265856"]
Jul 17 13:32:31.274801 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:32:31.274775    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbee8a3b9_5e3a_4629_ae9d_3140fafbf420.slice/crio-a3b1ad39f7e5d2d7f109c96f59cd6bf9c494ec5c93e963fc9658475f47aa74da WatchSource:0}: Error finding container a3b1ad39f7e5d2d7f109c96f59cd6bf9c494ec5c93e963fc9658475f47aa74da: Status 404 returned error can't find the container with id a3b1ad39f7e5d2d7f109c96f59cd6bf9c494ec5c93e963fc9658475f47aa74da
Jul 17 13:32:31.516266 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:31.516200    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-140520326265856" event={"ID":"bee8a3b9-5e3a-4629-ae9d-3140fafbf420","Type":"ContainerStarted","Data":"a3b1ad39f7e5d2d7f109c96f59cd6bf9c494ec5c93e963fc9658475f47aa74da"}
Jul 17 13:32:31.517572 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:31.517551    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326265856"
Jul 17 13:32:31.517661 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:31.517590    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91024515c1ee523e6bf1b0875a9b330043907b5ff109960304c76c8a4d083188"
Jul 17 13:32:31.529251 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:31.529230    2565 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7684-140520326265856" oldPodUID="4006fab5-7995-4072-826d-b5afa535ad0e" podUID="bee8a3b9-5e3a-4629-ae9d-3140fafbf420"
Jul 17 13:32:32.407799 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:32.407760    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4006fab5-7995-4072-826d-b5afa535ad0e" path="/var/lib/kubelet/pods/4006fab5-7995-4072-826d-b5afa535ad0e/volumes"
Jul 17 13:32:32.525270 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:32.525236    2565 generic.go:356] "Generic (PLEG): container finished" podID="bee8a3b9-5e3a-4629-ae9d-3140fafbf420" containerID="d49cd614c101de1bc7f9b710cd8de25afbddfc34a99024a7e2202371c5331e9d" exitCode=0
Jul 17 13:32:32.525446 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:32.525318    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-140520326265856" event={"ID":"bee8a3b9-5e3a-4629-ae9d-3140fafbf420","Type":"ContainerDied","Data":"d49cd614c101de1bc7f9b710cd8de25afbddfc34a99024a7e2202371c5331e9d"}
Jul 17 13:32:33.671805 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:33.671786    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326265856"
Jul 17 13:32:33.694106 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:33.694085    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7684-140520326265856_bee8a3b9-5e3a-4629-ae9d-3140fafbf420/test-curl-7684-140520326265856/0.log"
Jul 17 13:32:33.719164 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:33.719139    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7684-140520326265856"]
Jul 17 13:32:33.722848 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:33.722829    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7684-140520326265856"]
Jul 17 13:32:33.804627 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:33.804603    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpz8v\" (UniqueName: \"kubernetes.io/projected/bee8a3b9-5e3a-4629-ae9d-3140fafbf420-kube-api-access-vpz8v\") pod \"bee8a3b9-5e3a-4629-ae9d-3140fafbf420\" (UID: \"bee8a3b9-5e3a-4629-ae9d-3140fafbf420\") "
Jul 17 13:32:33.806693 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:33.806674    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee8a3b9-5e3a-4629-ae9d-3140fafbf420-kube-api-access-vpz8v" (OuterVolumeSpecName: "kube-api-access-vpz8v") pod "bee8a3b9-5e3a-4629-ae9d-3140fafbf420" (UID: "bee8a3b9-5e3a-4629-ae9d-3140fafbf420"). InnerVolumeSpecName "kube-api-access-vpz8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 13:32:33.906215 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:33.906186    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpz8v\" (UniqueName: \"kubernetes.io/projected/bee8a3b9-5e3a-4629-ae9d-3140fafbf420-kube-api-access-vpz8v\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:32:34.403647 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:34.403615    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bee8a3b9-5e3a-4629-ae9d-3140fafbf420" path="/var/lib/kubelet/pods/bee8a3b9-5e3a-4629-ae9d-3140fafbf420/volumes"
Jul 17 13:32:34.432273 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:34.432252    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:32:34.536957 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:34.536926    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:32:34.539589 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:34.539573    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-140520326265856"
Jul 17 13:32:34.539702 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:34.539626    2565 scope.go:117] "RemoveContainer" containerID="d49cd614c101de1bc7f9b710cd8de25afbddfc34a99024a7e2202371c5331e9d"
Jul 17 13:32:36.822555 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:36.822527    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 13:32:36.827827 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:36.827801    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 13:32:36.842266 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:36.842248    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 13:32:36.847148 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:36.847130    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-trg5s_ed8c8549-e57b-4306-b02b-7d1f466f7b4e/ovn-acl-logging/0.log"
Jul 17 13:32:39.396827 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:39.396792    2565 scope.go:117] "RemoveContainer" containerID="151816836cb4f8efa89fa7ed97f32592f6afb1018cb00b558b5c70cbcc2afb9d"
Jul 17 13:32:40.579787 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:40.579759    2565 generic.go:356] "Generic (PLEG): container finished" podID="8201c6fb-124c-4c58-a36f-0b036a8f2efb" containerID="4fe28aca3fda47764452155bb2dbfd74c25201e1782d5a667e74bcd5de66ed4d" exitCode=6
Jul 17 13:32:40.580219 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:40.579838    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr" event={"ID":"8201c6fb-124c-4c58-a36f-0b036a8f2efb","Type":"ContainerDied","Data":"4fe28aca3fda47764452155bb2dbfd74c25201e1782d5a667e74bcd5de66ed4d"}
Jul 17 13:32:40.580219 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:40.579904    2565 scope.go:117] "RemoveContainer" containerID="151816836cb4f8efa89fa7ed97f32592f6afb1018cb00b558b5c70cbcc2afb9d"
Jul 17 13:32:40.580325 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:40.580272    2565 scope.go:117] "RemoveContainer" containerID="4fe28aca3fda47764452155bb2dbfd74c25201e1782d5a667e74bcd5de66ed4d"
Jul 17 13:32:40.580467 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:32:40.580449    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr_odh-ai-gateway-infra(8201c6fb-124c-4c58-a36f-0b036a8f2efb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr" podUID="8201c6fb-124c-4c58-a36f-0b036a8f2efb"
Jul 17 13:32:41.610809 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:41.610773    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr"]
Jul 17 13:32:41.759389 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:41.759367    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr"
Jul 17 13:32:41.876952 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:41.876882    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8201c6fb-124c-4c58-a36f-0b036a8f2efb-maas-api-service-ca\") pod \"8201c6fb-124c-4c58-a36f-0b036a8f2efb\" (UID: \"8201c6fb-124c-4c58-a36f-0b036a8f2efb\") "
Jul 17 13:32:41.877303 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:41.877271    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8201c6fb-124c-4c58-a36f-0b036a8f2efb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8201c6fb-124c-4c58-a36f-0b036a8f2efb" (UID: "8201c6fb-124c-4c58-a36f-0b036a8f2efb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:32:41.977828 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:41.977804    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8201c6fb-124c-4c58-a36f-0b036a8f2efb-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:32:42.595958 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:42.595934    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr"
Jul 17 13:32:42.596147 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:42.595927    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr" event={"ID":"8201c6fb-124c-4c58-a36f-0b036a8f2efb","Type":"ContainerDied","Data":"3424c97a5c272709a556c7636804ab8760b957bca893636b7219aa5d101255a1"}
Jul 17 13:32:42.596147 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:42.596074    2565 scope.go:117] "RemoveContainer" containerID="4fe28aca3fda47764452155bb2dbfd74c25201e1782d5a667e74bcd5de66ed4d"
Jul 17 13:32:42.613815 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:42.613792    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr"]
Jul 17 13:32:42.619664 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:42.619644    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-d42qr"]
Jul 17 13:32:44.377875 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:44.377828    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e92eab-29738250-chrr4"]
Jul 17 13:32:44.379730 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:44.379707    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:32:44.381269 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:44.381249    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e92eab-29738250-chrr4"]
Jul 17 13:32:44.403602 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:44.403574    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8201c6fb-124c-4c58-a36f-0b036a8f2efb" path="/var/lib/kubelet/pods/8201c6fb-124c-4c58-a36f-0b036a8f2efb/volumes"
Jul 17 13:32:44.403932 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:44.403912    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb5e0ac8-e141-4e9f-a2c1-a75e874f1deb" path="/var/lib/kubelet/pods/cb5e0ac8-e141-4e9f-a2c1-a75e874f1deb/volumes"
Jul 17 13:32:49.007745 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:49.007709    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:32:55.075821 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:55.075780    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:32:55.154476 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:55.154451    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:32:59.710328 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:59.710241    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:32:59.946408 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:32:59.946378    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:33:00.696695 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:00.696661    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:33:00.859350 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:00.859318    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:33:10.604504 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:10.604453    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:33:10.610199 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:10.610171    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5c3765-29738250-7l6b7"]
Jul 17 13:33:10.614082 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:10.614053    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-5c3765-29738250-7l6b7"]
Jul 17 13:33:12.403528 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:12.403498    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80f0de93-703d-48eb-b7cb-aaeb6f9cc570" path="/var/lib/kubelet/pods/80f0de93-703d-48eb-b7cb-aaeb6f9cc570/volumes"
Jul 17 13:33:15.143759 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:15.143725    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:33:20.630658 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:20.630622    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl"]
Jul 17 13:33:20.631064 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:20.631051    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bee8a3b9-5e3a-4629-ae9d-3140fafbf420" containerName="test-curl-7684-140520326265856"
Jul 17 13:33:20.631064 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:20.631065    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bee8a3b9-5e3a-4629-ae9d-3140fafbf420" containerName="test-curl-7684-140520326265856"
Jul 17 13:33:20.631154 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:20.631075    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8201c6fb-124c-4c58-a36f-0b036a8f2efb" containerName="revoke-keys"
Jul 17 13:33:20.631154 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:20.631081    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8201c6fb-124c-4c58-a36f-0b036a8f2efb" containerName="revoke-keys"
Jul 17 13:33:20.631154 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:20.631087    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8201c6fb-124c-4c58-a36f-0b036a8f2efb" containerName="revoke-keys"
Jul 17 13:33:20.631154 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:20.631092    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8201c6fb-124c-4c58-a36f-0b036a8f2efb" containerName="revoke-keys"
Jul 17 13:33:20.631280 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:20.631168    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8201c6fb-124c-4c58-a36f-0b036a8f2efb" containerName="revoke-keys"
Jul 17 13:33:20.631280 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:20.631175    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8201c6fb-124c-4c58-a36f-0b036a8f2efb" containerName="revoke-keys"
Jul 17 13:33:20.631280 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:20.631182    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8201c6fb-124c-4c58-a36f-0b036a8f2efb" containerName="revoke-keys"
Jul 17 13:33:20.631280 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:20.631187    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bee8a3b9-5e3a-4629-ae9d-3140fafbf420" containerName="test-curl-7684-140520326265856"
Jul 17 13:33:20.634568 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:20.634553    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl"
Jul 17 13:33:20.637578 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:20.637553    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqj2m\""
Jul 17 13:33:20.641320 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:20.641295    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl"]
Jul 17 13:33:20.698968 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:20.698944    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d879ba9-30cc-468b-a80e-967c82b54751-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl\" (UID: \"2d879ba9-30cc-468b-a80e-967c82b54751\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl"
Jul 17 13:33:20.799911 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:20.799878    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d879ba9-30cc-468b-a80e-967c82b54751-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl\" (UID: \"2d879ba9-30cc-468b-a80e-967c82b54751\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl"
Jul 17 13:33:20.800439 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:20.800420    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d879ba9-30cc-468b-a80e-967c82b54751-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl\" (UID: \"2d879ba9-30cc-468b-a80e-967c82b54751\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl"
Jul 17 13:33:20.946694 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:20.946637    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl"
Jul 17 13:33:21.084103 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:21.084077    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl"]
Jul 17 13:33:21.086340 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:33:21.086314    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d879ba9_30cc_468b_a80e_967c82b54751.slice/crio-7169241ac09d894e1b5e0fdf3263c1cbc0956cf5a6bc1ea4b53b13d769f0fb1b WatchSource:0}: Error finding container 7169241ac09d894e1b5e0fdf3263c1cbc0956cf5a6bc1ea4b53b13d769f0fb1b: Status 404 returned error can't find the container with id 7169241ac09d894e1b5e0fdf3263c1cbc0956cf5a6bc1ea4b53b13d769f0fb1b
Jul 17 13:33:21.159061 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:21.159008    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:33:21.256779 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:21.256709    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:33:21.844854 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:21.844823    2565 generic.go:356] "Generic (PLEG): container finished" podID="2d879ba9-30cc-468b-a80e-967c82b54751" containerID="8e3cbb3d263b00d2f953c79c17098e426581656344b945fec061e0df16f71677" exitCode=6
Jul 17 13:33:21.845207 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:21.844910    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl" event={"ID":"2d879ba9-30cc-468b-a80e-967c82b54751","Type":"ContainerDied","Data":"8e3cbb3d263b00d2f953c79c17098e426581656344b945fec061e0df16f71677"}
Jul 17 13:33:21.845207 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:21.844961    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl" event={"ID":"2d879ba9-30cc-468b-a80e-967c82b54751","Type":"ContainerStarted","Data":"7169241ac09d894e1b5e0fdf3263c1cbc0956cf5a6bc1ea4b53b13d769f0fb1b"}
Jul 17 13:33:21.845207 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:21.845116    2565 scope.go:117] "RemoveContainer" containerID="8e3cbb3d263b00d2f953c79c17098e426581656344b945fec061e0df16f71677"
Jul 17 13:33:22.853183 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:22.853149    2565 generic.go:356] "Generic (PLEG): container finished" podID="2d879ba9-30cc-468b-a80e-967c82b54751" containerID="3cc06c00aeddb8ecedec69188c77155484742280500df090a05f6cd2e4322f2f" exitCode=6
Jul 17 13:33:22.853585 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:22.853238    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl" event={"ID":"2d879ba9-30cc-468b-a80e-967c82b54751","Type":"ContainerDied","Data":"3cc06c00aeddb8ecedec69188c77155484742280500df090a05f6cd2e4322f2f"}
Jul 17 13:33:22.853585 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:22.853298    2565 scope.go:117] "RemoveContainer" containerID="8e3cbb3d263b00d2f953c79c17098e426581656344b945fec061e0df16f71677"
Jul 17 13:33:22.853585 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:22.853462    2565 scope.go:117] "RemoveContainer" containerID="3cc06c00aeddb8ecedec69188c77155484742280500df090a05f6cd2e4322f2f"
Jul 17 13:33:22.853726 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:33:22.853660    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl_odh-ai-gateway-infra(2d879ba9-30cc-468b-a80e-967c82b54751)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl" podUID="2d879ba9-30cc-468b-a80e-967c82b54751"
Jul 17 13:33:23.861837 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:23.861807    2565 scope.go:117] "RemoveContainer" containerID="3cc06c00aeddb8ecedec69188c77155484742280500df090a05f6cd2e4322f2f"
Jul 17 13:33:23.862280 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:33:23.862009    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl_odh-ai-gateway-infra(2d879ba9-30cc-468b-a80e-967c82b54751)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl" podUID="2d879ba9-30cc-468b-a80e-967c82b54751"
Jul 17 13:33:25.844112 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:25.844078    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:33:26.066256 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:26.066230    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:33:31.879308 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:31.879282    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-prsfj_2fa55000-c5fd-43ea-b4f6-f9759d917db7/manager/0.log"
Jul 17 13:33:31.992573 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:31.992540    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wv999_cb1aed0b-a75b-4d94-a525-d267307d5bce/manager/0.log"
Jul 17 13:33:32.119388 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:32.119367    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-85456b7f49-6mkfq_d7c0a4cb-c50f-488f-9758-bfac25466cd7/manager/0.log"
Jul 17 13:33:32.240416 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:32.240362    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-r5kf4_92d74ba2-23de-403e-9856-e54b19eb9c25/server/0.log"
Jul 17 13:33:32.491872 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:32.491808    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d8fb4667b-4prbs_d039365a-014f-4608-879d-63fcd59f19b3/copy-manifests/0.log"
Jul 17 13:33:32.498624 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:32.498603    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d8fb4667b-4prbs_d039365a-014f-4608-879d-63fcd59f19b3/manager/0.log"
Jul 17 13:33:33.128113 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:33.128052    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5cbc55679f-gw844_bcd7660c-d7f1-4d9d-81bf-c5c45ae9a660/maas-api/0.log"
Jul 17 13:33:33.238192 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:33.238172    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738220-ffqq8_9fed3c4c-ed5c-4976-ad2a-969a1a3c3d13/cleanup/0.log"
Jul 17 13:33:33.465875 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:33.465782    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738250-t7586_0491467e-4f8e-4913-991c-ec0ced8cb91c/cleanup/0.log"
Jul 17 13:33:33.574618 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:33.574566    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl_2d879ba9-30cc-468b-a80e-967c82b54751/revoke-keys/1.log"
Jul 17 13:33:33.690940 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:33.690920    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-771703-b4fbf_88ec52a1-dbc0-4c2e-9db0-e83c0694fe9a/revoke-keys/0.log"
Jul 17 13:33:34.042287 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:34.042259    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-ae44b3-nftxk_de4febd3-cd03-4771-a48b-85c61b3e5b5b/revoke-keys/0.log"
Jul 17 13:33:34.396831 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:34.396804    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-6ad1d9a1-4w52h_cb342c5c-709a-4a95-9aa5-95114a67acaa/revoke-keys/0.log"
Jul 17 13:33:34.865502 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:34.865471    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-8z89s_ee43e5a0-0e95-48d3-8972-4c6876457f50/postgres/0.log"
Jul 17 13:33:35.665225 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:35.665198    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft_488f355f-da9e-448e-b066-ed881e791765/util/0.log"
Jul 17 13:33:35.672261 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:35.672235    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft_488f355f-da9e-448e-b066-ed881e791765/pull/0.log"
Jul 17 13:33:35.679217 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:35.679192    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft_488f355f-da9e-448e-b066-ed881e791765/extract/0.log"
Jul 17 13:33:35.788757 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:35.788736    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg_2b80e493-a458-4093-9173-922689bce863/util/0.log"
Jul 17 13:33:35.796305 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:35.796289    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg_2b80e493-a458-4093-9173-922689bce863/pull/0.log"
Jul 17 13:33:35.803242 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:35.803217    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg_2b80e493-a458-4093-9173-922689bce863/extract/0.log"
Jul 17 13:33:35.912797 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:35.912770    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp_80e4bd69-feb8-496a-966e-6cfea38936e7/util/0.log"
Jul 17 13:33:35.919673 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:35.919605    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp_80e4bd69-feb8-496a-966e-6cfea38936e7/pull/0.log"
Jul 17 13:33:35.925868 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:35.925853    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp_80e4bd69-feb8-496a-966e-6cfea38936e7/extract/0.log"
Jul 17 13:33:36.036855 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:36.036829    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg_fd980ea9-8de2-47a5-ae4b-2e5b17764ff0/util/0.log"
Jul 17 13:33:36.043213 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:36.043194    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg_fd980ea9-8de2-47a5-ae4b-2e5b17764ff0/pull/0.log"
Jul 17 13:33:36.049067 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:36.049039    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg_fd980ea9-8de2-47a5-ae4b-2e5b17764ff0/extract/0.log"
Jul 17 13:33:36.292340 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:36.292241    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-g7qk6_c67bff4e-9316-4494-ab0b-3115a5b59e82/manager/0.log"
Jul 17 13:33:36.413474 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:36.413428    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-p5pwp_0863561d-bd8f-43e9-ad0c-9e862c8ea29d/manager/0.log"
Jul 17 13:33:36.530732 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:36.530711    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-4pvhs_f46a17f9-aba8-4257-b052-f5cf6f87d5cc/kuadrant-console-plugin/0.log"
Jul 17 13:33:36.669089 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:36.669073    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-ckqdg_31148710-faca-46f4-af71-e367aabe39e6/registry-server/0.log"
Jul 17 13:33:36.808780 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:36.808756    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-766lt_45c88802-f8b6-4983-a40d-39523cc5c217/manager/0.log"
Jul 17 13:33:37.039707 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:37.039636    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-h7q6d_4182c07e-dbcf-4223-8253-da988273a232/manager/0.log"
Jul 17 13:33:37.750720 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:37.750697    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bfcdd9cc7-psxhp_52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07/kube-auth-proxy/0.log"
Jul 17 13:33:37.992021 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:37.991974    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-62qbl_dfb10fec-52a0-4c07-a09b-5c4e6f7d52de/payload-pre-processing/0.log"
Jul 17 13:33:38.111322 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:38.111279    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-pbxkj_798c57d9-278e-4438-b436-a1e2f3280a8e/payload-processing/0.log"
Jul 17 13:33:38.396623 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:38.396540    2565 scope.go:117] "RemoveContainer" containerID="3cc06c00aeddb8ecedec69188c77155484742280500df090a05f6cd2e4322f2f"
Jul 17 13:33:38.567238 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:38.567213    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-9v8gt_94d7f364-bb60-42ee-b6a0-bd7535023b8e/main/0.log"
Jul 17 13:33:38.686575 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:38.686522    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-crf9d_1664b90e-7ff9-4901-ab0c-0df556f4f295/main/0.log"
Jul 17 13:33:38.964285 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:38.964206    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl" event={"ID":"2d879ba9-30cc-468b-a80e-967c82b54751","Type":"ContainerStarted","Data":"487fbd6730bb8f78d4d4990e2b49664a91772f2fc0f039768e61af2c2ee5ef4f"}
Jul 17 13:33:38.982059 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:38.982016    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl" podStartSLOduration=18.982002249 podStartE2EDuration="18.982002249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 13:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:33:38.979461646 +0000 UTC m=+4263.110846307" watchObservedRunningTime="2026-07-17 13:33:38.982002249 +0000 UTC m=+4263.113386928"
Jul 17 13:33:39.139691 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:39.139672    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-wmjq2_2796cbe3-9897-47e0-95c6-12adf98aceab/main/0.log"
Jul 17 13:33:39.972937 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:39.972909    2565 generic.go:356] "Generic (PLEG): container finished" podID="2d879ba9-30cc-468b-a80e-967c82b54751" containerID="487fbd6730bb8f78d4d4990e2b49664a91772f2fc0f039768e61af2c2ee5ef4f" exitCode=6
Jul 17 13:33:39.973319 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:39.972999    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl" event={"ID":"2d879ba9-30cc-468b-a80e-967c82b54751","Type":"ContainerDied","Data":"487fbd6730bb8f78d4d4990e2b49664a91772f2fc0f039768e61af2c2ee5ef4f"}
Jul 17 13:33:39.973319 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:39.973058    2565 scope.go:117] "RemoveContainer" containerID="3cc06c00aeddb8ecedec69188c77155484742280500df090a05f6cd2e4322f2f"
Jul 17 13:33:39.973549 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:39.973370    2565 scope.go:117] "RemoveContainer" containerID="487fbd6730bb8f78d4d4990e2b49664a91772f2fc0f039768e61af2c2ee5ef4f"
Jul 17 13:33:39.973609 ip-10-0-134-119 kubenswrapper[2565]: E0717 13:33:39.973566    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl_odh-ai-gateway-infra(2d879ba9-30cc-468b-a80e-967c82b54751)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl" podUID="2d879ba9-30cc-468b-a80e-967c82b54751"
Jul 17 13:33:39.989700 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:39.989676    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl"]
Jul 17 13:33:41.120128 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:41.120108    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl"
Jul 17 13:33:41.275101 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:41.275036    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d879ba9-30cc-468b-a80e-967c82b54751-maas-api-service-ca\") pod \"2d879ba9-30cc-468b-a80e-967c82b54751\" (UID: \"2d879ba9-30cc-468b-a80e-967c82b54751\") "
Jul 17 13:33:41.275367 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:41.275347    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d879ba9-30cc-468b-a80e-967c82b54751-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2d879ba9-30cc-468b-a80e-967c82b54751" (UID: "2d879ba9-30cc-468b-a80e-967c82b54751"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 13:33:41.376363 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:41.376330    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d879ba9-30cc-468b-a80e-967c82b54751-maas-api-service-ca\") on node \"ip-10-0-134-119.ec2.internal\" DevicePath \"\""
Jul 17 13:33:41.988280 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:41.988253    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl"
Jul 17 13:33:41.988468 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:41.988257    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl" event={"ID":"2d879ba9-30cc-468b-a80e-967c82b54751","Type":"ContainerDied","Data":"7169241ac09d894e1b5e0fdf3263c1cbc0956cf5a6bc1ea4b53b13d769f0fb1b"}
Jul 17 13:33:41.988468 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:41.988400    2565 scope.go:117] "RemoveContainer" containerID="487fbd6730bb8f78d4d4990e2b49664a91772f2fc0f039768e61af2c2ee5ef4f"
Jul 17 13:33:42.011794 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:42.011772    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl"]
Jul 17 13:33:42.019341 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:42.019323    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12f6c9d7-n2nxl"]
Jul 17 13:33:42.411707 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:42.411674    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d879ba9-30cc-468b-a80e-967c82b54751" path="/var/lib/kubelet/pods/2d879ba9-30cc-468b-a80e-967c82b54751/volumes"
Jul 17 13:33:42.710690 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:42.710622    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:33:44.362537 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:44.362507    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:33:44.451569 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:44.451545    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:33:47.859958 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:47.859929    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vqfdd_881c9321-df8f-45af-a283-2fe1306b33b5/global-pull-secret-syncer/0.log"
Jul 17 13:33:47.909706 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:47.909677    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log"
Jul 17 13:33:48.053732 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:48.053702    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-119.ec2.internal_9e932a7529a4a2a40c2cedf7c627d21a/haproxy/0.log"
Jul 17 13:33:52.846219 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:52.846191    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft_488f355f-da9e-448e-b066-ed881e791765/extract/0.log"
Jul 17 13:33:52.864125 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:52.864103    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft_488f355f-da9e-448e-b066-ed881e791765/util/0.log"
Jul 17 13:33:52.891844 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:52.891801    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759d8fft_488f355f-da9e-448e-b066-ed881e791765/pull/0.log"
Jul 17 13:33:52.922013 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:52.921977    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg_2b80e493-a458-4093-9173-922689bce863/extract/0.log"
Jul 17 13:33:52.956583 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:52.956557    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg_2b80e493-a458-4093-9173-922689bce863/util/0.log"
Jul 17 13:33:52.993957 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:52.993938    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j5zpg_2b80e493-a458-4093-9173-922689bce863/pull/0.log"
Jul 17 13:33:53.027904 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:53.027885    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp_80e4bd69-feb8-496a-966e-6cfea38936e7/extract/0.log"
Jul 17 13:33:53.048688 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:53.048669    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp_80e4bd69-feb8-496a-966e-6cfea38936e7/util/0.log"
Jul 17 13:33:53.070608 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:53.070590    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xlzzp_80e4bd69-feb8-496a-966e-6cfea38936e7/pull/0.log"
Jul 17 13:33:53.096821 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:53.096752    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg_fd980ea9-8de2-47a5-ae4b-2e5b17764ff0/extract/0.log"
Jul 17 13:33:53.126130 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:53.126110    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg_fd980ea9-8de2-47a5-ae4b-2e5b17764ff0/util/0.log"
Jul 17 13:33:53.158476 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:53.158458    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef14vsfg_fd980ea9-8de2-47a5-ae4b-2e5b17764ff0/pull/0.log"
Jul 17 13:33:53.595858 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:53.595832    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-g7qk6_c67bff4e-9316-4494-ab0b-3115a5b59e82/manager/0.log"
Jul 17 13:33:53.632118 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:53.632069    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-p5pwp_0863561d-bd8f-43e9-ad0c-9e862c8ea29d/manager/0.log"
Jul 17 13:33:53.663805 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:53.663785    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-4pvhs_f46a17f9-aba8-4257-b052-f5cf6f87d5cc/kuadrant-console-plugin/0.log"
Jul 17 13:33:53.705406 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:53.705377    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-ckqdg_31148710-faca-46f4-af71-e367aabe39e6/registry-server/0.log"
Jul 17 13:33:53.816867 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:53.816833    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-766lt_45c88802-f8b6-4983-a40d-39523cc5c217/manager/0.log"
Jul 17 13:33:53.916934 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:53.916857    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-h7q6d_4182c07e-dbcf-4223-8253-da988273a232/manager/0.log"
Jul 17 13:33:55.452442 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:55.452413    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6a1de2bb-19b0-4eb1-9b61-16010ec58da3/alertmanager/0.log"
Jul 17 13:33:55.472315 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:55.472298    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6a1de2bb-19b0-4eb1-9b61-16010ec58da3/config-reloader/0.log"
Jul 17 13:33:55.500788 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:55.500768    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6a1de2bb-19b0-4eb1-9b61-16010ec58da3/kube-rbac-proxy-web/0.log"
Jul 17 13:33:55.523844 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:55.523824    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6a1de2bb-19b0-4eb1-9b61-16010ec58da3/kube-rbac-proxy/0.log"
Jul 17 13:33:55.545647 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:55.545630    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6a1de2bb-19b0-4eb1-9b61-16010ec58da3/kube-rbac-proxy-metric/0.log"
Jul 17 13:33:55.573653 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:55.573636    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6a1de2bb-19b0-4eb1-9b61-16010ec58da3/prom-label-proxy/0.log"
Jul 17 13:33:55.610652 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:55.610634    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6a1de2bb-19b0-4eb1-9b61-16010ec58da3/init-config-reloader/0.log"
Jul 17 13:33:55.698820 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:55.698798    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-dqsfh_14b4f94d-7ff0-4416-aa79-c2ebb50a19ec/kube-state-metrics/0.log"
Jul 17 13:33:55.725728 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:55.725676    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-dqsfh_14b4f94d-7ff0-4416-aa79-c2ebb50a19ec/kube-rbac-proxy-main/0.log"
Jul 17 13:33:55.749947 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:55.749910    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-dqsfh_14b4f94d-7ff0-4416-aa79-c2ebb50a19ec/kube-rbac-proxy-self/0.log"
Jul 17 13:33:55.955578 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:55.955556    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4k2wr_feebc2f4-3dab-4a0e-93e1-7a39f18ea223/node-exporter/0.log"
Jul 17 13:33:55.979473 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:55.979395    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4k2wr_feebc2f4-3dab-4a0e-93e1-7a39f18ea223/kube-rbac-proxy/0.log"
Jul 17 13:33:56.002418 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:56.002400    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4k2wr_feebc2f4-3dab-4a0e-93e1-7a39f18ea223/init-textfile/0.log"
Jul 17 13:33:56.461132 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:56.461112    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-7vkc7_9b65fed5-8ade-412d-857f-0206503fd06a/prometheus-operator-admission-webhook/0.log"
Jul 17 13:33:56.636213 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:56.636133    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75df9fcb9d-lct87_4b898525-cf0d-473b-8b77-d477947facf0/thanos-query/0.log"
Jul 17 13:33:56.660226 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:56.660193    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75df9fcb9d-lct87_4b898525-cf0d-473b-8b77-d477947facf0/kube-rbac-proxy-web/0.log"
Jul 17 13:33:56.685011 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:56.684975    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75df9fcb9d-lct87_4b898525-cf0d-473b-8b77-d477947facf0/kube-rbac-proxy/0.log"
Jul 17 13:33:56.707624 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:56.707607    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75df9fcb9d-lct87_4b898525-cf0d-473b-8b77-d477947facf0/prom-label-proxy/0.log"
Jul 17 13:33:56.730824 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:56.730807    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75df9fcb9d-lct87_4b898525-cf0d-473b-8b77-d477947facf0/kube-rbac-proxy-rules/0.log"
Jul 17 13:33:56.754190 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:56.754171    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-75df9fcb9d-lct87_4b898525-cf0d-473b-8b77-d477947facf0/kube-rbac-proxy-metrics/0.log"
Jul 17 13:33:57.911727 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:57.911700    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-qn4xk_2ea9caca-0266-4894-ad2e-c02b4e5eeddf/networking-console-plugin/0.log"
Jul 17 13:33:58.237209 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.237135    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7"]
Jul 17 13:33:58.237564 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.237551    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d879ba9-30cc-468b-a80e-967c82b54751" containerName="revoke-keys"
Jul 17 13:33:58.237612 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.237566    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d879ba9-30cc-468b-a80e-967c82b54751" containerName="revoke-keys"
Jul 17 13:33:58.237612 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.237583    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d879ba9-30cc-468b-a80e-967c82b54751" containerName="revoke-keys"
Jul 17 13:33:58.237612 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.237589    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d879ba9-30cc-468b-a80e-967c82b54751" containerName="revoke-keys"
Jul 17 13:33:58.237612 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.237595    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8201c6fb-124c-4c58-a36f-0b036a8f2efb" containerName="revoke-keys"
Jul 17 13:33:58.237612 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.237600    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8201c6fb-124c-4c58-a36f-0b036a8f2efb" containerName="revoke-keys"
Jul 17 13:33:58.237767 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.237665    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d879ba9-30cc-468b-a80e-967c82b54751" containerName="revoke-keys"
Jul 17 13:33:58.237767 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.237674    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d879ba9-30cc-468b-a80e-967c82b54751" containerName="revoke-keys"
Jul 17 13:33:58.237767 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.237681    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d879ba9-30cc-468b-a80e-967c82b54751" containerName="revoke-keys"
Jul 17 13:33:58.241063 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.241043    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7"
Jul 17 13:33:58.243888 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.243869    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8mj6w\"/\"openshift-service-ca.crt\""
Jul 17 13:33:58.245024 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.245005    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8mj6w\"/\"default-dockercfg-lsxbf\""
Jul 17 13:33:58.245106 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.245069    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8mj6w\"/\"kube-root-ca.crt\""
Jul 17 13:33:58.252232 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.252207    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7"]
Jul 17 13:33:58.316177 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.316153    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5857e731-7b42-40f5-8be9-1a76ba990f98-lib-modules\") pod \"perf-node-gather-daemonset-rmqt7\" (UID: \"5857e731-7b42-40f5-8be9-1a76ba990f98\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7"
Jul 17 13:33:58.316284 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.316193    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5857e731-7b42-40f5-8be9-1a76ba990f98-sys\") pod \"perf-node-gather-daemonset-rmqt7\" (UID: \"5857e731-7b42-40f5-8be9-1a76ba990f98\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7"
Jul 17 13:33:58.316284 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.316257    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5857e731-7b42-40f5-8be9-1a76ba990f98-podres\") pod \"perf-node-gather-daemonset-rmqt7\" (UID: \"5857e731-7b42-40f5-8be9-1a76ba990f98\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7"
Jul 17 13:33:58.316359 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.316284    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6786\" (UniqueName: \"kubernetes.io/projected/5857e731-7b42-40f5-8be9-1a76ba990f98-kube-api-access-p6786\") pod \"perf-node-gather-daemonset-rmqt7\" (UID: \"5857e731-7b42-40f5-8be9-1a76ba990f98\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7"
Jul 17 13:33:58.316359 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.316339    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5857e731-7b42-40f5-8be9-1a76ba990f98-proc\") pod \"perf-node-gather-daemonset-rmqt7\" (UID: \"5857e731-7b42-40f5-8be9-1a76ba990f98\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7"
Jul 17 13:33:58.416792 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.416766    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5857e731-7b42-40f5-8be9-1a76ba990f98-proc\") pod \"perf-node-gather-daemonset-rmqt7\" (UID: \"5857e731-7b42-40f5-8be9-1a76ba990f98\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7"
Jul 17 13:33:58.416914 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.416800    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5857e731-7b42-40f5-8be9-1a76ba990f98-lib-modules\") pod \"perf-node-gather-daemonset-rmqt7\" (UID: \"5857e731-7b42-40f5-8be9-1a76ba990f98\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7"
Jul 17 13:33:58.416914 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.416834    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5857e731-7b42-40f5-8be9-1a76ba990f98-sys\") pod \"perf-node-gather-daemonset-rmqt7\" (UID: \"5857e731-7b42-40f5-8be9-1a76ba990f98\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7"
Jul 17 13:33:58.416914 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.416868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5857e731-7b42-40f5-8be9-1a76ba990f98-podres\") pod \"perf-node-gather-daemonset-rmqt7\" (UID: \"5857e731-7b42-40f5-8be9-1a76ba990f98\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7"
Jul 17 13:33:58.416914 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.416887    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6786\" (UniqueName: \"kubernetes.io/projected/5857e731-7b42-40f5-8be9-1a76ba990f98-kube-api-access-p6786\") pod \"perf-node-gather-daemonset-rmqt7\" (UID: \"5857e731-7b42-40f5-8be9-1a76ba990f98\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7"
Jul 17 13:33:58.416914 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.416901    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5857e731-7b42-40f5-8be9-1a76ba990f98-proc\") pod \"perf-node-gather-daemonset-rmqt7\" (UID: \"5857e731-7b42-40f5-8be9-1a76ba990f98\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7"
Jul 17 13:33:58.417191 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.416936    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5857e731-7b42-40f5-8be9-1a76ba990f98-lib-modules\") pod \"perf-node-gather-daemonset-rmqt7\" (UID: \"5857e731-7b42-40f5-8be9-1a76ba990f98\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7"
Jul 17 13:33:58.417191 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.417009    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5857e731-7b42-40f5-8be9-1a76ba990f98-sys\") pod \"perf-node-gather-daemonset-rmqt7\" (UID: \"5857e731-7b42-40f5-8be9-1a76ba990f98\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7"
Jul 17 13:33:58.417191 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.417038    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5857e731-7b42-40f5-8be9-1a76ba990f98-podres\") pod \"perf-node-gather-daemonset-rmqt7\" (UID: \"5857e731-7b42-40f5-8be9-1a76ba990f98\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7"
Jul 17 13:33:58.425669 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.425644    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6786\" (UniqueName: \"kubernetes.io/projected/5857e731-7b42-40f5-8be9-1a76ba990f98-kube-api-access-p6786\") pod \"perf-node-gather-daemonset-rmqt7\" (UID: \"5857e731-7b42-40f5-8be9-1a76ba990f98\") " pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7"
Jul 17 13:33:58.552574 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.552553    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7"
Jul 17 13:33:58.558181 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.558163    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/1.log"
Jul 17 13:33:58.563274 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.563256    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-p8vsw_3b0956d7-a5e7-4710-8063-dbc0e5429f9f/console-operator/2.log"
Jul 17 13:33:58.686845 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:58.686810    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7"]
Jul 17 13:33:58.688277 ip-10-0-134-119 kubenswrapper[2565]: W0717 13:33:58.688248    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5857e731_7b42_40f5_8be9_1a76ba990f98.slice/crio-3f7925049bcb645c9217489e274cae3f525fcd0df19c40d05c4aeb82972ebe56 WatchSource:0}: Error finding container 3f7925049bcb645c9217489e274cae3f525fcd0df19c40d05c4aeb82972ebe56: Status 404 returned error can't find the container with id 3f7925049bcb645c9217489e274cae3f525fcd0df19c40d05c4aeb82972ebe56
Jul 17 13:33:59.093298 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:59.093232    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7" event={"ID":"5857e731-7b42-40f5-8be9-1a76ba990f98","Type":"ContainerStarted","Data":"bf1f2ad46e0adbc97cd6bd5f0f096c2831b2eda9d4b4d55960a3411e7d9f5d54"}
Jul 17 13:33:59.093298 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:59.093280    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7" event={"ID":"5857e731-7b42-40f5-8be9-1a76ba990f98","Type":"ContainerStarted","Data":"3f7925049bcb645c9217489e274cae3f525fcd0df19c40d05c4aeb82972ebe56"}
Jul 17 13:33:59.093696 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:59.093327    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7"
Jul 17 13:33:59.111146 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:33:59.111108    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7" podStartSLOduration=1.111095642 podStartE2EDuration="1.111095642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 13:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 13:33:59.10791809 +0000 UTC m=+4283.239302752" watchObservedRunningTime="2026-07-17 13:33:59.111095642 +0000 UTC m=+4283.242480304"
Jul 17 13:34:00.738617 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:00.738590    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gbnmx_793d61b2-520e-416a-b856-545c0a0f45b5/dns/0.log"
Jul 17 13:34:00.760233 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:00.760199    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gbnmx_793d61b2-520e-416a-b856-545c0a0f45b5/kube-rbac-proxy/0.log"
Jul 17 13:34:00.897675 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:00.897652    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kw5vk_d0be7fb8-4dda-4b00-9b21-6c105e5bff69/dns-node-resolver/0.log"
Jul 17 13:34:01.480860 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:01.480831    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b25ws_de956422-3a00-4360-87d9-e43b2cb93859/node-ca/0.log"
Jul 17 13:34:02.645513 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:02.645485    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bfcdd9cc7-psxhp_52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07/kube-auth-proxy/0.log"
Jul 17 13:34:02.737100 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:02.737074    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-62qbl_dfb10fec-52a0-4c07-a09b-5c4e6f7d52de/payload-pre-processing/0.log"
Jul 17 13:34:02.763656 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:02.763630    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-pbxkj_798c57d9-278e-4438-b436-a1e2f3280a8e/payload-processing/0.log"
Jul 17 13:34:03.492665 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:03.492619    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n7hp4_02906c20-b77c-4c91-b4e4-fcd72c647670/serve-healthcheck-canary/0.log"
Jul 17 13:34:03.995720 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:03.995683    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8qwp2_97f2ceb2-b1ef-4380-b3ce-8e9ff2498282/kube-rbac-proxy/0.log"
Jul 17 13:34:04.016335 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:04.016317    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8qwp2_97f2ceb2-b1ef-4380-b3ce-8e9ff2498282/exporter/0.log"
Jul 17 13:34:04.040461 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:04.040444    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8qwp2_97f2ceb2-b1ef-4380-b3ce-8e9ff2498282/extractor/0.log"
Jul 17 13:34:05.109693 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:05.109667    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8mj6w/perf-node-gather-daemonset-rmqt7"
Jul 17 13:34:06.338838 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:06.338802    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-prsfj_2fa55000-c5fd-43ea-b4f6-f9759d917db7/manager/0.log"
Jul 17 13:34:06.459343 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:06.459320    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wv999_cb1aed0b-a75b-4d94-a525-d267307d5bce/manager/0.log"
Jul 17 13:34:06.557820 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:06.557790    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-85456b7f49-6mkfq_d7c0a4cb-c50f-488f-9758-bfac25466cd7/manager/0.log"
Jul 17 13:34:06.580906 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:06.580880    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-r5kf4_92d74ba2-23de-403e-9856-e54b19eb9c25/server/0.log"
Jul 17 13:34:06.691486 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:06.691419    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d8fb4667b-4prbs_d039365a-014f-4608-879d-63fcd59f19b3/manager/0.log"
Jul 17 13:34:06.711348 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:06.711330    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d8fb4667b-4prbs_d039365a-014f-4608-879d-63fcd59f19b3/copy-manifests/0.log"
Jul 17 13:34:08.327882 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:08.327860    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-fk64p_5f7f96d6-0e5f-4da3-8fea-db93ff820da5/openshift-lws-operator/0.log"
Jul 17 13:34:08.977127 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:08.977098    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bfcdd9cc7-psxhp_52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07/kube-auth-proxy/0.log"
Jul 17 13:34:09.054138 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:09.054116    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-62qbl_dfb10fec-52a0-4c07-a09b-5c4e6f7d52de/payload-pre-processing/0.log"
Jul 17 13:34:09.074438 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:09.074414    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-pbxkj_798c57d9-278e-4438-b436-a1e2f3280a8e/payload-processing/0.log"
Jul 17 13:34:10.488442 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:10.488412    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bfcdd9cc7-psxhp_52b6c1e2-fad9-4ca4-a5b4-b38102dcdc07/kube-auth-proxy/0.log"
Jul 17 13:34:10.572947 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:10.572923    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-62qbl_dfb10fec-52a0-4c07-a09b-5c4e6f7d52de/payload-pre-processing/0.log"
Jul 17 13:34:10.595923 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:10.595902    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-pbxkj_798c57d9-278e-4438-b436-a1e2f3280a8e/payload-processing/0.log"
Jul 17 13:34:14.292952 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:14.292918    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b6hrc_c97f4e9b-753e-4429-b477-03de51ee3fde/konnectivity-agent/0.log" limit=1024
Jul 17 13:34:14.972176 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:14.972151    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nb225_264b5b0f-2f3e-4acf-af1b-9d731a914efc/kube-multus-additional-cni-plugins/0.log"
Jul 17 13:34:14.994234 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:14.994207    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nb225_264b5b0f-2f3e-4acf-af1b-9d731a914efc/egress-router-binary-copy/0.log"
Jul 17 13:34:15.013017 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:15.012971    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nb225_264b5b0f-2f3e-4acf-af1b-9d731a914efc/cni-plugins/0.log"
Jul 17 13:34:15.032286 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:15.032269    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nb225_264b5b0f-2f3e-4acf-af1b-9d731a914efc/bond-cni-plugin/0.log"
Jul 17 13:34:15.052548 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:15.052529    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nb225_264b5b0f-2f3e-4acf-af1b-9d731a914efc/routeoverride-cni/0.log"
Jul 17 13:34:15.073397 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:15.073374    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nb225_264b5b0f-2f3e-4acf-af1b-9d731a914efc/whereabouts-cni-bincopy/0.log"
Jul 17 13:34:15.096892 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:15.096872    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nb225_264b5b0f-2f3e-4acf-af1b-9d731a914efc/whereabouts-cni/0.log"
Jul 17 13:34:15.339642 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:15.339619    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-sqgfk_eeee8e18-1099-4f80-b580-96a3d6af05a5/kube-multus/0.log"
Jul 17 13:34:15.491770 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:15.491745    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ml8cr_e5f413a7-be80-43f3-950e-be9b5c4bf9b4/network-metrics-daemon/0.log"
Jul 17 13:34:15.510450 ip-10-0-134-119 kubenswrapper[2565]: I0717 13:34:15.510429    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ml8cr_e5f413a7-be80-43f3-950e-be9b5c4bf9b4/kube-rbac-proxy/0.log"